# Check candidate correctness Generated: 2026-09-13T04:41:35.943Z Source build: local Canonical docs: https://teammately.ai/docs --- id: integrations.check-candidate-correctness title: Check candidate correctness summary: Use Teammately benchmark evidence to compare a candidate behavior change against a baseline before internal human review. kind: recipe product_area: benchmark_evaluations status: stable updated: 2026-08-23 canonical: /docs/integrations/check-candidate-correctness --- # Check candidate correctness This page belongs with Benchmark Evaluations because it uses benchmark runs, comparisons, failures, and coverage gaps to prepare evidence for internal human review. ## When to use this check Use this when a candidate behavior change already has comparable benchmark runs and the team needs human-readable context before an internal reviewer decides what to do next. ## Operating pattern 1. Confirm the baseline and candidate runs use the intended benchmark version and run metadata. 2. Inspect comparison results, must-level rubric failures, incomplete evidence, and coverage gaps. 3. Separate improvements from regressions by policy, rubric, case segment, or benchmark dimension. 4. Identify failures that require expert judgment before the candidate can be trusted. 5. Prepare review context from Benchmark Evaluations results without claiming Teammately records the final approval. 6. Link reviewers to the relevant run results, comparison view, and failing case details. {% example-demo title="Candidate review" %} Raw case: A team compares the current assistant with a candidate assistant before human review. Expert judgment: Downstream action should depend on approved standards, representative coverage, and explainable failures. Policy: Must-level policies are hard gates; prefer-level policies are quality signals for tradeoff review. Applicability: Applies to benchmark cases included in the approved benchmark version. Binary rubric: Each applicable rubric produces pass, fail, or incomplete evidence for the candidate output. Benchmark result: The candidate improves grounding but fails several must-level compatibility rubrics. Human review context: If the customer assembles a review packet, it names the exact Benchmark Version and saved Harness Version and separates compatibility failures from unresolved coverage gaps. {% /example-demo %} ## Evidence to collect - Baseline run, candidate run, benchmark version, and run metadata. - Comparison results for must-level rubrics, incomplete evidence, regressions, and improvements. - Failing cases, relevant policies, applicability logic, and rubric results. - Coverage gaps or missing-output states that limit confidence. - Review notes that explain what Teammately evidence covers and what remains a human decision. ## Related docs {% related-card-grid title="Related docs" %} - [Read run results](/docs/benchmark-evaluations/inspect-results) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Run a benchmark](/docs/benchmark-evaluations/run-evaluation) {% /related-card-grid %} ## Source confidence Code-backed: this page is grounded in benchmark evaluations routes and types. It does not claim a code-backed downstream approval workflow. --- id: coverage.benchmark-snapshots title: Benchmark snapshots summary: Freeze a benchmark into a version so every run measures the same cases and judgment rules. kind: concept product_area: coverage_engineering status: stable updated: 2026-08-23 canonical: /docs/coverage-engineering/benchmark-snapshots --- # Benchmark snapshots ## Definition A Benchmark Dataset Snapshot is the immutable evidence boundary created from a benchmark's current editable dataset. It records the selected Cases, eligible Policy and Rubric relationships, representation facts, label, content hash, Case count, and creation time used to establish a Benchmark Version. The live dataset can continue changing after the Snapshot exists. Those later changes do not rewrite an earlier Snapshot or any Run that used its Benchmark Version. ## Why it matters Evaluation results are comparable only when readers know which evidence boundary each Run used. Without the Snapshot identity, a score can move because the Harness changed, because Case membership changed, because evaluator relationships changed, or because representation shifted. Freezing the dataset keeps those explanations separate. ## Where it appears in the product Use **Benchmark Datasets → Snapshots** to run readiness checks, create the immutable Snapshot, and inspect its read-only Cases and Representation. Use Coverage Management to improve gaps before snapshotting. Use Benchmark Evaluations after the resulting Benchmark Version is available. ## Artifacts it affects A Snapshot binds selected Case versions, admitted evaluator relationships, and representation facts. It does not freeze the entire project, lock the current dataset, approve weak Cases, or prove that coverage is sufficient. Project Cases, Policies, Rubrics, and Coverage Facets remain reusable objects with their own lifecycle. Create a new Snapshot when a change affects the claim an evaluation will support—for example Case membership, relevant Case content or materials, admitted evaluator relationships, or the represented behavior boundary. A label-only correction that does not change evidence should not be described as new behavioral coverage. ## Operational check Before creating a Snapshot, confirm that readiness has no blockers, Cases are interpretable, required materials are ready, evaluator links are eligible and approved, and Representation supports the intended benchmark claim. After creation, record the Snapshot or Benchmark Version identity in every Run, comparison, and Improvement Session that depends on it. {% example-demo title="Preserving a coverage expansion" %} A source-authority Coverage Story produces eight reviewed Cases covering conflicts between current and superseded documents. The team adds those Cases to the current dataset and confirms their Rubric relationships. Earlier Runs remain attached to the old Benchmark Version. The operator creates a new Snapshot and uses its resulting version for the next evaluation, making the coverage change explicit instead of presenting the score movement as a Harness-only change. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Benchmarks and versions](/docs/concepts/benchmarks-and-versions) - [Create a Dataset Snapshot](/docs/benchmark-datasets/snapshots) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Product quickstart](/docs/quickstart) - [Task index](/docs/operating-manual/task-index) {% /related-card-grid %} ## Source confidence Code-backed: the active Snapshot manager and benchmark-scoped Snapshots route define readiness, immutable Snapshot identity, content hashes, Case counts, and read-only inspection. Coverage sufficiency remains a human interpretation supported by Representation and Coverage Management. --- id: benchmark-evaluations.run title: Run a Benchmark Evaluation summary: Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/run-evaluation --- # Run a Benchmark Evaluation Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready. ## Prerequisites - A Benchmark Version backed by the intended Dataset Snapshot. - Approved eligible Policies and Rubrics. - At least one saved project Harness Version. - Prepared Harness runtime and required secret grants. - A chosen number of Runs for each selected Harness. ## Steps 1. Open **Benchmark Evaluations** for the intended Benchmark Version. 2. Open Evaluation Settings if you need to adjust the machine configuration. 3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone. 4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume. 5. Supply any requested Run Metadata. Keep credentials out of descriptive fields. 6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one. 7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores. 8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable. ## Evidence created The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage. Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete. > No Draft execution > > A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching. {% example-demo title="Example: two candidates, three attempts" %} Harness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort. {% /example-demo %} ## Object and state changes Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort. ## Success criteria - Exact Harness and Benchmark Versions are recorded. - Each launch group contains the number of Runs requested for that Harness. - Output and evaluation progress reach an interpretable terminal state. - Incomplete or failed attempts remain visible. ## Common failure modes - Selecting the wrong saved Version or assuming Draft execution. - Reading partial evaluation as a complete cohort. - Treating absent telemetry as zero usage. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings) - [Harnesses](/docs/assets/harnesses) - [Run Metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Missing outputs](/docs/troubleshooting/missing-outputs) - [Authentication](/docs/troubleshooting/authentication) {% /related-card-grid %} ## Source confidence Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity. --- id: benchmark-evaluations.compare title: Compare Harness Versions summary: Compare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/compare --- # Compare Harness Versions Compare shows aggregate observed evidence for saved Harness Versions. Choose the Versions, compatible evaluation configuration, and measurement to compare. Use **Inspect individual Run comparisons** for the detailed result matrix. ## Prerequisites Saved Harness Versions with evaluation results are shown for the selected Benchmark Version. Choose one or more Versions to inspect; Run counts may differ. An output-only imported reference Run cannot become a Harness column because it has no executable saved Version. ## Select a row mode The individual Run matrix offers row modes including Cases, all results, Policies, Rubrics, Dimension ontology values, Project Topics, Topic Groups, and Case Construction Patterns. Use Cases to inspect concrete disagreement, Policies or Rubrics to locate correctness movement, and Coverage Facets to see whether gains concentrate in one behavior slice. ## Steps 1. Confirm the immutable Benchmark Version and choose at least two visible Harness Versions. 2. Select an aggregate measurement, or open individual Run comparisons and choose a row mode. 3. Check evidence completeness for each Harness column. A blank or incomplete cell is not a failure. 4. Locate rows with material disagreement and connect them back to Case and evaluator evidence. 5. Preserve regressions and required-criterion failures next to gains. 6. Use the exact candidate and row evidence when starting an Improvement Session or requesting an Expert Contribution. Compare reads existing evidence and does not mutate Runs. Selecting aggregate Harness Versions recomputes their comparison on compatible evidence; individual matrix visibility is local presentation. It does not activate Harnesses or choose a winner. {% example-demo title="Example: facet-local improvement" %} Three Harness Versions look similar overall. The Project Topic row mode shows that Version 14 improves source-authority Topics but regresses escalation Topics. Switching to Cases identifies two regressions, and the team starts Improve with those exact failures instead of claiming a uniform improvement. {% /example-demo %} ## Common mistakes - Comparing different Benchmark Versions as though only the candidate moved. - Treating missing evidence as a failed cell. - Reading a facet aggregate without checking the distinct Cases behind it. - Describing an imported output-only Run as a Harness Version. - Selecting the newest Version solely because it is newest. ## Object and state changes Compare reads existing evidence. Selecting aggregate Versions reads their observed evidence; selecting individual matrix row modes changes presentation; it does not activate a Harness, mutate a Run, or retain a candidate. ## Success criteria - At least two exact Harness Versions share the same Benchmark Version. - Incomplete cells remain distinct from failed evidence. - Material movement resolves to Cases, evaluators, or Coverage Facets. ## Common failure modes - Comparing moved evidence boundaries as candidate-only change. - Treating local column visibility as product configuration. - Using an output-only Run as a Harness column. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Harnesses](/docs/assets/harnesses) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} ## Source confidence Code-backed: the active Compare route and Evaluation Matrix define Harness columns, local visibility, symmetric comparison, and the current evidence row modes. --- id: improve.overview title: Improve summary: Coordinate durable Improvement Sessions from pinned benchmark evidence through Goal Contracts, candidates, evaluations, trajectories, and frontiers. kind: concept product_area: improve status: stable updated: 2026-09-07 canonical: /docs/improve --- # Improve Improve is the benchmark-scoped workspace for coordinating candidate work from benchmark evidence. Each Improvement Session pins the exact starting evidence it uses, including any Benchmark Version, Harness Version, and Run identities. Sessions preserve the goal, authorization, agent and worker activity, candidate Harness Versions, canonical evaluation receipts, and Current frontier. ## Definition A new session begins with **Start from evidence**. The operator selects the target and relevant benchmark evidence, then prepares a **Goal Contract**. The contract binds the free-form objective to canonical target identities, measurement bindings, constraints, unresolved items, and intervention boundaries. The current Improve workspace offers **Coevolve**, **External Agents**, and **External fine-tuning** experiences where enabled. Coevolve supports bounded Work and Evolve behavior; external experiences use explicit handoff or provider boundaries. All retained candidates still rely on observable Harness Versions and canonical Benchmark Evaluations rather than unrecorded claims of progress. ## Decision checkpoint | Need | Session choice | Required evidence | | --- | --- | --- | | Investigate or implement one bounded direction | Work | Confirmed Goal Contract and pinned failures or target evidence | | Explore several competing hypotheses | Evolve | Comparable measurement plan and explicit exploration authorization | | Delegate implementation externally | Worker package | Pinned target, return contract, and observable returned candidate | | Decide which candidates remain viable | Current frontier | Canonical evaluation receipts and goal constraints | | Stop scheduling new work temporarily | Pause | Retained evidence and chronology remain available | | Correct missing benchmark meaning | Contribution or coverage follow-up | Upstream policy, rubric, case, or coverage observation is named | ## Goal Contract and chronology The Goal Contract is the session's authority boundary. Confirm it only after the target and measurement plan are resolvable. Later revisions should remain explicit rather than silently changing what earlier candidates attempted. Chronology records durable session transitions, agent activity, proposed approaches, candidates, evaluations, pauses, resumptions, cancellation, and terminal state. Narrated trajectories expose bounded work segments and can identify unannounced durable activity without pretending to know private external-worker behavior. ## Candidates and current frontier A candidate should identify its Harness version, hypothesis, parent or branch context, and evaluation evidence. The current frontier represents retained candidates supported by the latest valid evidence and Goal Contract. It is not synonymous with the most recent candidate or the highest aggregate score. Evaluation receipts link candidates back to Benchmark Evaluations. If a candidate improves one objective but violates a constraint or regresses a must-level rubric, the frontier should keep that tradeoff visible. ## Human and external-worker boundary Operators can pause, resume, cancel, review, or redirect work through the available session controls. External worker packages can carry the confirmed goal and evidence to Codex, Claude Code, or another worker. Teammately records only returned artifacts and requests that satisfy the return contract; it does not infer private work state. {% example-demo title="Evolve a grounding harness" %} The Goal Contract targets unsupported blends of current and superseded sources without regressing missing-source uncertainty. Evolve explores retrieval filtering, source ranking, and response-construction branches. Each candidate Harness version is evaluated against the pinned benchmark. The current frontier retains the candidates supported by both grounding and uncertainty evidence, while a newly discovered policy conflict becomes an Expert Contribution opportunity. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Write and confirm a Goal Contract](/docs/improve/goal-contracts) - [Choose Work or Evolve](/docs/improve/work-and-evolve) - [Interpret candidates and the current frontier](/docs/improve/candidates-and-frontier) - [Read chronology and trajectories](/docs/improve/chronology-and-trajectories) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coevolve](/docs/concepts/coevolve) - [Harnesses](/docs/assets/harnesses) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Code-backed: the active versioned Improve workspace and session contract define starting evidence, Goal Contracts, Work and Evolve behavior, candidates, receipts, chronology, frontiers, and lifecycle controls. Internal coordinator implementation is supporting evidence, not a public API.