# Prepare human review context Generated: 2026-09-13T04:42:42.743Z Source build: local Canonical docs: https://teammately.ai/docs --- id: operating.prepare-review-packet title: Prepare human review context summary: Assemble customer-owned review context from exact evaluation, contribution, coverage, and improvement evidence. kind: task product_area: operating_manual status: stable updated: 2026-08-22 canonical: /docs/operating-manual/prepare-review-packet --- # Prepare human review context Assemble review context when an accountable customer team needs to inspect what the benchmark evidence says, why it says it, and which uncertainty or follow-up remains. This is a customer-owned packet or process, not a separate Teammately product object. ## Prerequisites - Completed or clearly bounded Benchmark Evaluation evidence. - Exact benchmark, dataset snapshot, Harness, Run, settings, and metadata identities. - Relevant Expert Contributions and governed policies or rubrics. - Coverage and Improvement Session context where it affects interpretation. ## Steps 1. State the review question and the downstream owner without implying that Teammately makes the final decision. 2. Identify the exact benchmark version, dataset snapshot, candidate Harness version, Runs, settings, and Run Metadata. 3. Summarize overall movement, then list material case-level gains, regressions, and uncertainty. 4. Link each important conclusion to applicable policies, rubrics, cases, and expert provenance. 5. Include relevant coverage gaps or representation limits. 6. Describe Improvement Session candidates and frontier evidence without claiming unobserved external-worker activity. 7. Separate confirmed findings, unresolved correctness, missing evidence, and recommended next investigation. 8. Preserve the source links or identifiers another reviewer needs to reproduce the interpretation. ## Object and state changes Preparing context should read existing Teammately artifacts rather than mutate them. Follow-up work may create a Contribution, policy or rubric revision, Coverage Story, Case, dataset snapshot, Run, or Improvement Session. Keep the reviewed evidence unchanged so the reason for follow-up remains available. ## Success criteria - Every conclusion is traceable to exact product evidence. - Aggregate results are supported by case and rubric detail. - Coverage limitations and unresolved expert disagreement are explicit. - Historical and current candidate boundaries are distinguishable. - The customer-owned downstream decision is not represented as a Teammately state. ## Common failure modes - Copying a score without versions and settings. - Omitting must-level regressions because the average improved. - Treating an AI summary or trajectory as expert authority. - Hiding missing coverage or unresolved source conflict. - Describing a downstream choice as if Teammately automatically made it. {% example-demo title="Example: review context for a retrieval change" %} The packet names the two saved Harness versions, benchmark snapshot, grounding and uncertainty rubrics, and compared Runs. It highlights improved current-source cases, regressed missing-source cases, one unresolved expert contribution, and the Improvement Session frontier. The accountable team can inspect the evidence and decide its own next action. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Expert Contributions](/docs/expert-contributions) - [Improve](/docs/improve) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Low expert agreement](/docs/troubleshooting/low-expert-agreement) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines the customer-owned human review boundary using current product artifacts without inventing a dedicated review-packet object or downstream-decision workflow. --- id: benchmark-evaluations.overview title: Benchmark Evaluations summary: Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare. kind: concept product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations --- # Benchmark Evaluations Benchmark Evaluations is the version-scoped workspace for executing and comparing candidate systems. The active top-level tabs are **Dashboard**, **List**, **Arena**, and **Compare**. Every managed Run binds an exact saved Harness Version to the immutable Benchmark Version shown in the route. > Evaluation boundary > > Interpret evidence inside its recorded Benchmark Version, Harness Version, Run or Run Group, evaluator set, and metadata. Run counts belong to launches. Additional launches add evidence without rewriting earlier Runs. ## Surfaces and objects Dashboard summarizes progress, leaderboards, rank progression across Runs, and available resource telemetry. List is segmented into **Runs**, **Evaluation results**, and **Traces / Spans**. The results segment summarizes Case outcomes and Policy or Rubric failures. Arena compares candidate pairs across governed metrics. Compare is a symmetric matrix of Harness Versions across selected evidence rows. A Run Group can collect one standard attempt or repeated attempts. A Run records one candidate execution and its per-Case progress. Evaluation results record the admitted Policy and Rubric outcomes. Costs, tokens, and latency are telemetry only when the provider or execution path captured them. > Traces / Spans capability fence > > The List navigation exposes Traces / Spans, but the current benchmark API does not expose evaluation execution traces. Do not claim that trajectories, spans, private reasoning, or tool traces can be inspected from Benchmark Evaluations today. ## Decision checkpoint | Need | Open | Evidence to preserve | | --- | --- | --- | | Configure and launch managed Runs | Evaluation Settings and New evaluation run | Machine, saved Harness Versions, and per-Harness Run counts | | Start candidate execution | Run modal | Exact Harness and Benchmark Versions | | Inspect status and output summaries | List → Runs or Evaluation results | Run Group, attempt, Case counts, incomplete state | | Compare candidate pairs | Arena | Metric family, pair count, only-A, only-B, shared failures | | Compare many candidates by governed rows | Compare | Harness columns and chosen Case or facet row mode | | Admit external reference outputs | Output mapping | Case mapping, attempt assignment, insert/update report | ## Rankings and repeated sampling Dashboard aggregates compatible observed Runs for each saved Harness Version across launches. Average score weights Runs equally. Supported binary views report passed at least once or passed every time over the observed case outcomes. Counts and missing evidence are shown; unequal counts do not prevent comparison. Historical group metrics retain their recorded meanings. Ranking is a routing signal. A candidate can lead overall while failing required Policy or high-impact Rubric evidence. Use Arena or Compare to locate the disagreement and List to confirm completeness before starting Improve work. ## External outputs Uploaded or API-supplied reference outputs create output-only Runs that can be scored and inspected in List. They are not saved Harness Versions and therefore cannot be optimized in Improve or selected as Harness columns in Compare or Arena. {% example-demo title="Example: repeated evaluation without evidence drift" %} A team launches three Runs of Harness Version 8 and one Run of Version 11 against the same Benchmark Version. Both appear with their evidence counts. A later launch of Version 11 adds two Runs to its aggregate evidence without changing either launch group. The team can inspect individual Runs before deciding whether more evidence is useful. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Configure evaluation execution](/docs/benchmark-evaluations/execution-settings) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Use Arena and rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) {% /related-card-grid %} ## Source confidence Code-backed: the active version-scoped workspace, settings, Run modal, List segments, Dashboard, Arena, and Compare routes define the current evaluation model and capability fences. --- id: expert-contributions.overview title: Expert Contributions summary: Coordinate benchmark-scoped expert work, attributable judgment, governed artifacts, and the decisions that move correctness forward. kind: concept product_area: expert_contributions status: stable updated: 2026-09-07 canonical: /docs/expert-contributions --- # Expert Contributions Expert Contributions is the benchmark-scoped workspace for requesting, conducting, and materializing specialist work. It coordinates the expert, objective, selected evidence, task sequence, checkpoints, attributable responses, and contributed artifacts needed to move a benchmark forward. ## Definition The administrator workspace contains **Overview**, **Contributions**, **Contributed Artifacts**, and **Logs & Status**. **Request Contribution** opens the composer for a new contribution. The expert follows a contribution-specific experience that can contain form, chat, interview, and case-review tasks, along with checkpoints and completion states. A Contribution is the unit of requested expert effort. It replaces broad workflow configuration with a bounded statement of what this benchmark needs from this expert now. The work can result in contributed policies, rubrics, cases, or coverage observations without flattening all expert activity into one generic approval record. ## Decision checkpoint | Need | Contribution element | Result to inspect | | --- | --- | --- | | Resolve a specific benchmark question | Contribution statement and scoped objectives | The expert can explain the requested decision | | Ground work in concrete behavior | Selected or designated cases | Case-level responses remain attributable | | Supply supporting knowledge | Attachments and scoped statements | The expert sees the relevant source boundary | | Choose the right interaction | Form, chat, interview, or case review task | Task output matches the kind of judgment needed | | Confirm consequential learning | Checkpoint | Accepted, revised, or unresolved state is explicit | | Reuse the result | Contributed Artifacts | Policies, rubrics, cases, and coverage observations retain provenance | ## Lifecycle and status The durable Contribution statuses are `PREPARING_DIRECTION`, `AWAITING_DIRECTION_ALIGNMENT`, `MATERIALIZING_TASKS`, `READY`, `IN_PROGRESS`, `COMPLETED`, and `CANCELLED`. The interface presents these as planning direction, waiting for alignment, preparing tasks, ready, active, completed, or cancelled. The exact task sequence can vary by Contribution. Realtime updates and durable transitions help the administrator and expert see current progress without inventing completion. A waiting state, checkpoint, or finalization step should be shown as such. Completing the expert experience does not imply that every proposed artifact has been accepted into its project-level owner. ## Contribution evidence Logs & Status exposes operational and engagement records. Contributed Artifacts organizes materialized or contributed cases, policies, rubrics, and new coverage observations. Correctness Governance, Assets, or Coverage Management owns the resulting project or benchmark artifact after materialization. This model improves return on expert effort. Agents prepare focused work from project context, indexed material, benchmark cases, and unresolved questions. The expert supplies the authority; the result can be reused across standards, coverage, evaluation, and improvement. {% example-demo title="Resolve source authority" %} A benchmark contains cases where an operational runbook conflicts with a newer policy page. The operator requests a Contribution from the policy owner, selects the conflicting cases, attaches both sources, and uses case review plus a checkpoint. The expert establishes which source controls, contributes a scoped policy and rubric, and records one coverage observation for an unrepresented exception. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) - [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution) - [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Contributed Artifacts](/docs/expert-contributions/contributed-artifacts) - [Contribution lifecycle and status](/docs/expert-contributions/lifecycle-and-status) - [Logs & Status](/docs/expert-contributions/logs-and-status) - [Agent Setup](/docs/agent-setup) - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) {% /related-card-grid %} ## Source confidence Code-backed: the active benchmark workspace, Contribution dashboard, composer, administrator detail, and expert routes support the scope, task, status, and artifact model described here. --- id: improve.overview title: Improve summary: Coordinate durable Improvement Sessions from pinned benchmark evidence through Goal Contracts, candidates, evaluations, trajectories, and frontiers. kind: concept product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve --- # Improve Improve is the benchmark-scoped workspace for coordinating candidate work from benchmark evidence. Each Improvement Session pins the exact starting evidence it uses, including any Benchmark Version, Harness Version, and Run identities. Sessions preserve the goal, authorization, agent and worker activity, candidate Harness Versions, canonical evaluation receipts, and Current frontier. ## Definition A new session begins with **Start from evidence**. The operator selects the target and relevant benchmark evidence, then prepares a **Goal Contract**. The contract binds the free-form objective to canonical target identities, measurement bindings, constraints, unresolved items, and intervention boundaries. The current Improve workspace offers **Coevolve**, **External Agents**, and **External fine-tuning** experiences where enabled. Coevolve supports bounded Work and Evolve behavior; external experiences use explicit handoff or provider boundaries. All retained candidates still rely on observable Harness Versions and canonical Benchmark Evaluations rather than unrecorded claims of progress. ## Decision checkpoint | Need | Session choice | Required evidence | | --- | --- | --- | | Investigate or implement one bounded direction | Work | Confirmed Goal Contract and pinned failures or target evidence | | Explore several competing hypotheses | Evolve | Comparable measurement plan and explicit exploration authorization | | Delegate implementation externally | Worker package | Pinned target, return contract, and observable returned candidate | | Decide which candidates remain viable | Current frontier | Canonical evaluation receipts and goal constraints | | Stop scheduling new work temporarily | Pause | Retained evidence and chronology remain available | | Correct missing benchmark meaning | Contribution or coverage follow-up | Upstream policy, rubric, case, or coverage observation is named | ## Goal Contract and chronology The Goal Contract is the session's authority boundary. Confirm it only after the target and measurement plan are resolvable. Later revisions should remain explicit rather than silently changing what earlier candidates attempted. Chronology records durable session transitions, agent activity, proposed approaches, candidates, evaluations, pauses, resumptions, cancellation, and terminal state. Narrated trajectories expose bounded work segments and can identify unannounced durable activity without pretending to know private external-worker behavior. ## Candidates and current frontier A candidate should identify its Harness version, hypothesis, parent or branch context, and evaluation evidence. The current frontier represents retained candidates supported by the latest valid evidence and Goal Contract. It is not synonymous with the most recent candidate or the highest aggregate score. Evaluation receipts link candidates back to Benchmark Evaluations. If a candidate improves one objective but violates a constraint or regresses a must-level rubric, the frontier should keep that tradeoff visible. ## Human and external-worker boundary Operators can pause, resume, cancel, review, or redirect work through the available session controls. External worker packages can carry the confirmed goal and evidence to Codex, Claude Code, or another worker. Teammately records only returned artifacts and requests that satisfy the return contract; it does not infer private work state. {% example-demo title="Evolve a grounding harness" %} The Goal Contract targets unsupported blends of current and superseded sources without regressing missing-source uncertainty. Evolve explores retrieval filtering, source ranking, and response-construction branches. Each candidate Harness version is evaluated against the pinned benchmark. The current frontier retains the candidates supported by both grounding and uncertainty evidence, while a newly discovered policy conflict becomes an Expert Contribution opportunity. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Write and confirm a Goal Contract](/docs/improve/goal-contracts) - [Choose Work or Evolve](/docs/improve/work-and-evolve) - [Interpret candidates and the current frontier](/docs/improve/candidates-and-frontier) - [Read chronology and trajectories](/docs/improve/chronology-and-trajectories) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coevolve](/docs/concepts/coevolve) - [Harnesses](/docs/assets/harnesses) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Code-backed: the active versioned Improve workspace and session contract define starting evidence, Goal Contracts, Work and Evolve behavior, candidates, receipts, chronology, frontiers, and lifecycle controls. Internal coordinator implementation is supporting evidence, not a public API.