# Improve Generated: 2026-09-13T04:33:12.736Z Source build: local Canonical docs: https://teammately.ai/docs --- id: improve.overview title: Improve summary: Coordinate durable Improvement Sessions from pinned benchmark evidence through Goal Contracts, candidates, evaluations, trajectories, and frontiers. kind: concept product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve --- # Improve Improve is the benchmark-scoped workspace for coordinating candidate work from benchmark evidence. Each Improvement Session pins the exact starting evidence it uses, including any Benchmark Version, Harness Version, and Run identities. Sessions preserve the goal, authorization, agent and worker activity, candidate Harness Versions, canonical evaluation receipts, and Current frontier. ## Definition A new session begins with **Start from evidence**. The operator selects the target and relevant benchmark evidence, then prepares a **Goal Contract**. The contract binds the free-form objective to canonical target identities, measurement bindings, constraints, unresolved items, and intervention boundaries. The current Improve workspace offers **Coevolve**, **External Agents**, and **External fine-tuning** experiences where enabled. Coevolve supports bounded Work and Evolve behavior; external experiences use explicit handoff or provider boundaries. All retained candidates still rely on observable Harness Versions and canonical Benchmark Evaluations rather than unrecorded claims of progress. ## Decision checkpoint | Need | Session choice | Required evidence | | --- | --- | --- | | Investigate or implement one bounded direction | Work | Confirmed Goal Contract and pinned failures or target evidence | | Explore several competing hypotheses | Evolve | Comparable measurement plan and explicit exploration authorization | | Delegate implementation externally | Worker package | Pinned target, return contract, and observable returned candidate | | Decide which candidates remain viable | Current frontier | Canonical evaluation receipts and goal constraints | | Stop scheduling new work temporarily | Pause | Retained evidence and chronology remain available | | Correct missing benchmark meaning | Contribution or coverage follow-up | Upstream policy, rubric, case, or coverage observation is named | ## Goal Contract and chronology The Goal Contract is the session's authority boundary. Confirm it only after the target and measurement plan are resolvable. Later revisions should remain explicit rather than silently changing what earlier candidates attempted. Chronology records durable session transitions, agent activity, proposed approaches, candidates, evaluations, pauses, resumptions, cancellation, and terminal state. Narrated trajectories expose bounded work segments and can identify unannounced durable activity without pretending to know private external-worker behavior. ## Candidates and current frontier A candidate should identify its Harness version, hypothesis, parent or branch context, and evaluation evidence. The current frontier represents retained candidates supported by the latest valid evidence and Goal Contract. It is not synonymous with the most recent candidate or the highest aggregate score. Evaluation receipts link candidates back to Benchmark Evaluations. If a candidate improves one objective but violates a constraint or regresses a must-level rubric, the frontier should keep that tradeoff visible. ## Human and external-worker boundary Operators can pause, resume, cancel, review, or redirect work through the available session controls. External worker packages can carry the confirmed goal and evidence to Codex, Claude Code, or another worker. Teammately records only returned artifacts and requests that satisfy the return contract; it does not infer private work state. {% example-demo title="Evolve a grounding harness" %} The Goal Contract targets unsupported blends of current and superseded sources without regressing missing-source uncertainty. Evolve explores retrieval filtering, source ranking, and response-construction branches. Each candidate Harness version is evaluated against the pinned benchmark. The current frontier retains the candidates supported by both grounding and uncertainty evidence, while a newly discovered policy conflict becomes an Expert Contribution opportunity. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Write and confirm a Goal Contract](/docs/improve/goal-contracts) - [Choose Work or Evolve](/docs/improve/work-and-evolve) - [Interpret candidates and the current frontier](/docs/improve/candidates-and-frontier) - [Read chronology and trajectories](/docs/improve/chronology-and-trajectories) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coevolve](/docs/concepts/coevolve) - [Harnesses](/docs/assets/harnesses) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Code-backed: the active versioned Improve workspace and session contract define starting evidence, Goal Contracts, Work and Evolve behavior, candidates, receipts, chronology, frontiers, and lifecycle controls. Internal coordinator implementation is supporting evidence, not a public API. --- id: improve.candidates-frontier title: Candidates and the Current Frontier summary: Interpret candidate stages, canonical receipts, constraints, retained Candidate Systems, and the current frontier. kind: reference product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve/candidates-and-frontier --- # Candidates and the Current Frontier ## Definition An Improve candidate moves through `working`, `focused_evaluated`, `canonical_evaluated`, `retained`, `rejected`, or `incomparable`. Stage describes evidence maturity, not source quality or recency. ## Fields, states, or lifecycle rules Candidate stage, Harness Version, Benchmark Version, canonical Run, provenance, scores, constraint results, parent branch, and receipts form the durable evidence record. Frontier membership is a retained evidence decision rather than another candidate stage. ## Candidate System requirements A retained Candidate System identifies an exact saved Harness Version, immutable Benchmark Version, canonical Run, provenance, scores, and constraint results. Focused evaluation can route exploration, but only canonical evaluation against the pinned cohort supports frontier retention. A rejected candidate remains useful evidence when it reveals a regression or failed hypothesis. An incomparable candidate lacks the common evidence boundary needed for a valid decision. Do not fill that gap with aggregate intuition. Candidate Exploration evidence is scoped to the session and branch that produced it. It is not automatically the project's global best candidate. The current frontier contains retained candidates supported by the latest valid Goal and receipts, but “current” does not mean newest or highest aggregate score. ## Harness materialization boundary The active Improve model does not require a separate legacy “Save candidate as Harness” action. A candidate eligible for retention already references an immutable saved Harness Version. Use Assets → Harnesses to inspect that Version or select it directly in a future Benchmark Evaluation launch. If an external worker returns only a patch, prose, or private claim, the candidate is not yet a Candidate System. Materialize the return through the supported contract so the saved Version and evaluation request become observable. ## Retention checklist - Exact Harness and Benchmark Versions are present. - The canonical Run is complete enough for the measurement bindings. - Hard constraints pass; soft-constraint tradeoffs are visible. - Evaluation receipts and hashes resolve to the claimed Case cohort. - Parent branch and source provenance are recorded. - The candidate is retained for evidence, not merely because it is latest. {% example-demo title="Example: aggregate winner outside the frontier" %} A new candidate has the highest overall pass rate but violates a hard required-Policy constraint. Its canonical receipt remains visible, but the candidate is rejected and does not displace two lower-scoring retained candidates that satisfy the confirmed Goal. {% /example-demo %} ## Related task pages {% related-card-grid title="Related task pages" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) {% /related-card-grid %} ## Source confidence Code-backed: the active session contract, archive view, and coordinator evidence define candidate stages, exact identities, canonical receipts, retention, rejection, incomparability, and frontier semantics. --- id: improve.chronology-trajectories title: Chronology, Trajectories, and Receipts summary: Read durable session events, safe narrated work segments, evaluation receipts, external handoffs, and usage evidence. kind: reference product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve/chronology-and-trajectories --- # Chronology, Trajectories, and Receipts ## Definition Chronology is the durable ordered record of an Improvement Session. Event blocks can represent user or assistant messages, activity, Harness output, usage, Goal changes, input requests, external handoffs, and cancellation. Pagination preserves long sessions without implying that the initially loaded page is complete history. ## Fields, states, or lifecycle rules - Chronology event types retain their identity and ordering. - Trajectory segments link only to observable activity. - Evaluation ledger rows preserve Case counts, completion, receipts, and hashes. - Worker packages and sessions have independent lifecycle states. ## Safe trajectories Trajectory segments narrate observable work and link to durable activity references. They may explain that a candidate was prepared, evaluated, retained, or rejected. They exclude hidden reasoning and private worker state. Do not rewrite silence between returned artifacts as a detailed external-worker trajectory. Benchmark Evaluations does not currently expose Traces / Spans; Improve trajectories are a separate safe session narration surface. They should not be described as raw model reasoning, execution spans, or evaluator authority. ## Evaluation ledger and receipts The evaluation ledger records exact Case counts, evaluable and incomplete populations, scores, constraint results, and receipt or content hashes. A receipt links a candidate claim to the canonical Benchmark Evaluation that supports it. When the cohort is incomplete, preserve that state in frontier and completion decisions. Usage entries can identify provider use by model. Missing usage evidence is unknown, not zero. Harness output records candidate-visible results without granting access to private runtime reasoning. ## External handoffs An external worker package is prepared only after a Goal Contract is confirmed. Its status can be prepared, submitted, expired, or closed. The return contract accepts an immutable Harness Version or a supported evaluation request. Expiration or closure describes the package lifecycle, not whether private work occurred. ## Session lifecycle Sessions can be draft, active, paused, completing, completed, cancelling, cancelled, or failed. Attention states identify an input or review need. Pausing stops new scheduling while retaining chronology and evidence. Cancelling records a terminal path; it does not erase candidates, receipts, or usage already recorded. > Observable-state rule > > Chronology may report only durable messages, events, returned artifacts, requests, and receipts. Never infer hidden chain of thought or fabricate progress for an external worker. ## Related task pages {% related-card-grid title="Related task pages" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Choose Work or Evolve](/docs/improve/work-and-evolve) {% /related-card-grid %} ## Source confidence Code-backed: chronology rendering, the session contract, and archive model define event types, safe trajectories, ledger and receipt fields, handoff states, usage, pagination, and lifecycle controls. --- id: improve.goal-contracts title: Goal Contracts summary: Bind an Improvement Session to exact targets, measurable objectives, constraints, and permitted intervention scope. kind: reference product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve/goal-contracts --- # Goal Contracts ## Definition A Goal Contract is the authority and measurement boundary for one Improvement Session. It turns a natural-language intent into canonical targets, prioritized objectives, protected constraints, and evaluable bindings. ## Fields, states, or lifecycle rules ### Contract contents The contract identifies exact target IDs and records intent. Objectives carry priorities. Constraints are hard or soft. Measurement bindings name whether evidence should **improve**, **preserve**, or **reduce** a canonical Case, Policy, Rubric, metric, or other admitted reference. Intervention constraints state what work may change. Unresolved items remain explicit until the contract can be confirmed safely. Every requested outcome needs an observable binding. “Make answers better” is not measurable; “improve the selected grounding Rubric while preserving the selected uncertainty Cases” is. A hard constraint disqualifies a candidate when violated. A soft constraint records a tradeoff that still requires review. ### Revisions and confirmation Goal Contract revisions can be proposed, confirmed, superseded, or rejected. Keep superseded and rejected revisions as history so candidate activity can be explained against the contract that authorized it. Do not silently edit the meaning of a session after work has begun. Evolve locks the Goal Contract after the first epoch begins. If new evidence reveals a fundamentally different goal, stop or complete the current session and create an explicit new boundary rather than retrofitting prior epochs. ### Confirmation checklist - Target Harness, Harness Version, Benchmark Version, and starting Run resolve to exact identities. - Each objective has a direction and canonical measurement binding. - Non-regression behavior is represented by preserve bindings or constraints. - Hard and soft constraints are distinguishable. - Intervention scope permits the intended code, prompt, retrieval, or configuration work. - No unresolved item makes evaluation or authorization ambiguous. > Authority boundary > > The Goal Contract authorizes session work; it does not change the Benchmark Version, approve a new correctness standard, or waive human governance of upstream artifacts. ## Related task pages {% related-card-grid title="Related task pages" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Choose Work or Evolve](/docs/improve/work-and-evolve) {% /related-card-grid %} ## Source confidence Code-backed: the active session contract and commands define canonical targets, objectives, constraints, measurement bindings, intervention scope, revision states, confirmation, and Evolve locking. --- id: improve.start-session title: Start an Improvement Session summary: Start from benchmark evidence, prepare a measurable Goal Contract, and choose bounded Work or Evolve behavior. kind: task product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve/start-improvement-session --- # Start an Improvement Session Start an Improvement Session when evaluation evidence justifies a candidate change or bounded investigation. The setup should turn a free-form intention into a measurable Goal Contract before work begins. Choose the saved Harness Version and a specific baseline launch. A baseline may contain one Run. Set the number of Runs for future candidate evaluations independently; a difference from the baseline count is informational. The work forecast and authorized budget use the chosen candidate count. ## Prerequisites - A selected benchmark and evidence that identifies the relevant Benchmark Version or Run. - An existing target Harness and saved starting version. - Case, rubric, Run, comparison, or frontier evidence that explains the need. - A measurable outcome and constraints that should remain protected. - An operator authorized to start and control the session. ## Steps 1. Open the selected benchmark and choose **Improve**. 2. Choose the available Improve experience, then create a new Improvement Session and select **Work** or **Evolve** when using Coevolve. 3. Select the target Harness, exact starting Harness Version, starting Run, and execution source. 4. For Work, choose the available Coevolve or External Agents path. For Evolve, use Coevolve and choose user-gated or autonomous execution. External fine-tuning has its own provider and return boundary when available. 5. State the desired behavior change and important non-regression constraints. 6. Prepare the Goal Contract. Resolve canonical target identities, objectives, measurement bindings, intervention constraints, and unresolved items. 7. Inspect the proposed revision and confirm it only when the evidence can measure the requested outcome. 8. For Evolve, configure epoch authorization, Case pass target, and provider usage bounds before starting. 9. If using an external worker, verify the scoped package and return contract after the Goal is confirmed. 10. Start the session and use chronology, trajectories, candidates, receipts, and current frontier to follow observable progress. ## Object and state changes This task creates a benchmark-scoped Improvement Session, records its mode, experience, source, target, and pinned starting evidence, and establishes a Goal Contract revision. Session lifecycle states are draft, active, paused, completing, completed, cancelling, cancelled, or failed, with attention states when operator action is needed. Starting work can create worker packages, candidate Harness Versions, canonical evaluation requests and receipts, frontier changes, chronology events, and usage records. ## Success criteria - The target and starting evidence use canonical identities. - Every objective has an observable measurement binding. - Constraints protect important behavior from hidden regression. - Work or Evolve is chosen deliberately. - The exact starting Harness Version and Run are visible. - Candidate progress is supported by returned artifacts and evaluation receipts. - The current frontier is explainable from the Goal Contract and evidence. ## Common failure modes - Starting from an aggregate score without selected case or rubric evidence. - Confirming a Goal Contract whose outcome cannot be measured. - Allowing Evolve without bounded authorization. - Treating a worker package as proof that private work occurred. - Retaining the newest candidate without checking constraints and regressions. - Changing the benchmark boundary during the session without making the new evidence explicit. {% example-demo title="Example: bounded Work session" %} A Run fails three cases because the Harness uses a superseded source. The operator pins those cases and the grounding rubric, targets the exact saved Harness version, and writes a Goal Contract requiring current-source selection without reducing missing-source uncertainty performance. Work begins only after both objectives have measurement bindings. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Improve](/docs/improve) - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Harnesses](/docs/assets/harnesses) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} ## Source confidence Code-backed: the current Improve setup, session command, and Goal Contract behavior support this workflow. External workers remain bounded by observable return artifacts and requests. --- id: improve.work-evolve title: Work and Evolve summary: Choose bounded implementation work or multi-branch evolutionary search with explicit epoch and provider authorization. kind: task product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve/work-and-evolve --- # Work and Evolve ## Prerequisites - A confirmed Goal Contract with exact starting evidence. - A saved target Harness Version and canonical measurement bindings. - Authorization for the selected worker source, epochs, and provider use. Within the Coevolve experience, choose **Work** for one bounded implementation or investigation and **Evolve** when the confirmed Goal supports systematic exploration across several candidate branches and epochs. External Agents and External fine-tuning use different execution boundaries and should be selected only when their return or provider contracts are clear. ## Work Work can run through Teammately or an offered external execution source such as Codex, Claude, or another worker. The source receives the confirmed Goal and scoped evidence. It must return an observable immutable Harness Version or evaluation request; a handoff package alone is not proof that private work occurred. Use Work when the likely intervention is known, the change is narrow, or operator review should follow one candidate at a time. Review returned source, candidate identity, evaluation receipt, constraint result, and chronology before treating the work as complete. ## Evolve Evolve uses Teammately and explores three Patch-to-Eval branches per epoch against the full pinned cohort. Choose **user-gated** to approve each epoch or **autonomous** to authorize a bounded number of epochs from 1 through 100. Set the target Case pass percentage and review provider usage by model. An epoch does not merely generate text. Each viable branch must materialize an exact saved Harness Version and obtain canonical evaluation evidence before retention. Failed, incomparable, or constraint-violating branches remain visible rather than being presented as improvement. ## Select the mode Use Work when evidence points to a specific retrieval filter, prompt rule, tool call, or output mapping change. Use Evolve when multiple independent interventions could satisfy the Goal and the benchmark can distinguish them. Do not use autonomous epochs when the Goal has unresolved authority, the evaluation boundary is unstable, or provider usage is not authorized. > Epoch authorization > > User-gated and autonomous execution change how future epochs are scheduled, not the acceptance criteria. Every retained candidate still needs exact identity, canonical evaluation, and Goal-constraint compliance. ## Object and state changes Starting Work or Evolve advances the Session and can create handoff packages, epoch branches, saved Harness Versions, evaluations, receipts, usage, and frontier decisions. Pause and cancellation preserve recorded evidence. ## Success criteria - Mode and execution source match the problem. - Every retained branch has exact candidate identity and canonical evidence. - Epoch limits, pass target, and provider usage remain within authorization. ## Common failure modes - Using Evolve before the benchmark can distinguish hypotheses. - Treating a handoff package as returned implementation. - Retaining a focused-only or constraint-violating candidate. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Goal Contracts](/docs/improve/goal-contracts) - [Candidates and the Current Frontier](/docs/improve/candidates-and-frontier) - [Chronology, Trajectories, and Receipts](/docs/improve/chronology-and-trajectories) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} ## Source confidence Code-backed: the active Improve setup, session contract, and work-review model define sources, Work and Evolve modes, authorization styles, epoch bounds, three branches, pinned cohort, pass target, and provider usage. --- id: concepts.coevolve title: Coevolve summary: Explore multiple evidence-backed candidate directions while keeping goals, benchmark receipts, trajectories, and the current frontier connected. kind: concept product_area: coevolve status: stable updated: 2026-08-22 canonical: /docs/concepts/coevolve --- # Coevolve Coevolve is the improvement capability inside Teammately correctness infrastructure. It lets agents and engineers pursue multiple candidate directions from benchmark evidence, evaluate those candidates through the same canonical path, and continue from stronger branches without losing the goal, chronology, or evidence behind the current frontier. ## Definition Coevolve is realized through **Improve** and durable **Improvement Sessions**. A session starts from selected benchmark evidence and a target Harness. A Goal Contract turns a free-form objective into pinned target identities, measurement bindings, constraints, unresolved items, and permitted intervention boundaries. Once confirmed, that contract governs future work without rewriting earlier evidence. Sessions can use Work or Evolve behavior. Work supports bounded implementation and review. Evolve supports parallel candidate exploration across epochs and retains a frontier based on current evaluation evidence. Candidate Harness versions become meaningful only when canonical Runs return observable results. ## Decision checkpoint | Situation | Improvement action | Evidence requirement | | --- | --- | --- | | One known candidate change needs implementation | Start a Work session | Confirmed Goal Contract and pinned target evidence | | Several hypotheses should compete | Start an Evolve session | Explicit authorization, measurement bindings, and comparable evaluation path | | An external coding worker will implement | Prepare a scoped worker package | Returned Harness version or evaluation request before claiming observable progress | | A candidate looks stronger | Inspect the current frontier | Canonical evaluation receipts support the retained position | | Exploration exposes missing correctness or coverage | Return the observation upstream | Identify the policy, rubric, case, or coverage artifact that must change | ## Evidence-backed branching Candidate exploration is not a sequence of undocumented edits. Each proposal should state the hypothesis and its relationship to the Goal Contract. Evaluation receipts bind candidate identity to benchmark identity and result. Narrated trajectories can explain the work performed, while chronology records durable transitions, agent activity, pauses, resumptions, and terminal state. The current frontier is not simply the newest candidate. It represents the candidates retained by the session's evidence and goal constraints. A candidate can improve one slice and regress another; the frontier and comparison views keep that tradeoff visible. ## Bridge between experts and engineers Improvement can reveal that the candidate is not the only incomplete part of the system. A missing coverage tuple, unclear rubric, contradictory policy, or insufficient case material should become an upstream contribution opportunity. This is how Coevolve connects coding agents to domain experts: engineering work is guided by benchmark evidence, and newly discovered correctness questions return to focused expert work. External workers remain bounded. Teammately may prepare a package for Codex, Claude Code, or another worker, but it records only the work returned through the defined contract. It does not infer private activity or fabricate a working state. {% example-demo title="Parallel grounding hypotheses" %} An Evolve session starts from failures involving conflicting policy documents. One candidate changes retrieval filtering, another changes source ranking, and a third changes answer construction. Each saved Harness version is evaluated against the pinned benchmark. The frontier retains the candidates supported by grounding and uncertainty rubrics, while a newly observed source-authority ambiguity becomes an Expert Contribution opportunity. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Improve](/docs/improve) - [Harnesses](/docs/assets/harnesses) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines Coevolve as the public capability. The Improve pages provide code-backed session, contract, candidate, and frontier behavior. --- id: benchmark-evaluations.overview title: Benchmark Evaluations summary: Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare. kind: concept product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations --- # Benchmark Evaluations Benchmark Evaluations is the version-scoped workspace for executing and comparing candidate systems. The active top-level tabs are **Dashboard**, **List**, **Arena**, and **Compare**. Every managed Run binds an exact saved Harness Version to the immutable Benchmark Version shown in the route. > Evaluation boundary > > Interpret evidence inside its recorded Benchmark Version, Harness Version, Run or Run Group, evaluator set, and metadata. Run counts belong to launches. Additional launches add evidence without rewriting earlier Runs. ## Surfaces and objects Dashboard summarizes progress, leaderboards, rank progression across Runs, and available resource telemetry. List is segmented into **Runs**, **Evaluation results**, and **Traces / Spans**. The results segment summarizes Case outcomes and Policy or Rubric failures. Arena compares candidate pairs across governed metrics. Compare is a symmetric matrix of Harness Versions across selected evidence rows. A Run Group can collect one standard attempt or repeated attempts. A Run records one candidate execution and its per-Case progress. Evaluation results record the admitted Policy and Rubric outcomes. Costs, tokens, and latency are telemetry only when the provider or execution path captured them. > Traces / Spans capability fence > > The List navigation exposes Traces / Spans, but the current benchmark API does not expose evaluation execution traces. Do not claim that trajectories, spans, private reasoning, or tool traces can be inspected from Benchmark Evaluations today. ## Decision checkpoint | Need | Open | Evidence to preserve | | --- | --- | --- | | Configure and launch managed Runs | Evaluation Settings and New evaluation run | Machine, saved Harness Versions, and per-Harness Run counts | | Start candidate execution | Run modal | Exact Harness and Benchmark Versions | | Inspect status and output summaries | List → Runs or Evaluation results | Run Group, attempt, Case counts, incomplete state | | Compare candidate pairs | Arena | Metric family, pair count, only-A, only-B, shared failures | | Compare many candidates by governed rows | Compare | Harness columns and chosen Case or facet row mode | | Admit external reference outputs | Output mapping | Case mapping, attempt assignment, insert/update report | ## Rankings and repeated sampling Dashboard aggregates compatible observed Runs for each saved Harness Version across launches. Average score weights Runs equally. Supported binary views report passed at least once or passed every time over the observed case outcomes. Counts and missing evidence are shown; unequal counts do not prevent comparison. Historical group metrics retain their recorded meanings. Ranking is a routing signal. A candidate can lead overall while failing required Policy or high-impact Rubric evidence. Use Arena or Compare to locate the disagreement and List to confirm completeness before starting Improve work. ## External outputs Uploaded or API-supplied reference outputs create output-only Runs that can be scored and inspected in List. They are not saved Harness Versions and therefore cannot be optimized in Improve or selected as Harness columns in Compare or Arena. {% example-demo title="Example: repeated evaluation without evidence drift" %} A team launches three Runs of Harness Version 8 and one Run of Version 11 against the same Benchmark Version. Both appear with their evidence counts. A later launch of Version 11 adds two Runs to its aggregate evidence without changing either launch group. The team can inspect individual Runs before deciding whether more evidence is useful. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Configure evaluation execution](/docs/benchmark-evaluations/execution-settings) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Use Arena and rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) {% /related-card-grid %} ## Source confidence Code-backed: the active version-scoped workspace, settings, Run modal, List segments, Dashboard, Arena, and Compare routes define the current evaluation model and capability fences.