# Benchmarks and versions Generated: 2026-09-13T04:39:38.138Z Source build: local Canonical docs: https://teammately.ai/docs --- id: concepts.benchmarks-versions title: Benchmarks and versions summary: Understand benchmarks, benchmark versions, snapshots, and why versioned measurement matters. kind: concept product_area: object_model status: stable updated: 2026-08-23 canonical: /docs/concepts/benchmarks-and-versions --- # Benchmarks and versions ## Definition A Benchmark is the durable workspace for one evaluation program. Its current Dataset and coverage work can change over time. A Dataset Snapshot freezes selected Case membership; a Benchmark Version fixes the evidence boundary used by an Evaluation Run. ## Why it matters This separation keeps evidence interpretable. A score can change because the saved Harness Version changed, the Dataset changed, or the governed evaluator boundary changed. Naming the Benchmark Version prevents those causes from being collapsed into “the model changed.” ## Version sanity check | Before comparing runs, confirm... | Otherwise... | | --- | --- | | The same benchmark version was used, or the version change is intentional and named. | Score movement may be caused by changed cases or standards. | | The Dataset Snapshot and governed evaluator boundary are identifiable. | A comparison can hide the evidence that actually changed. | | The saved Harness Version, settings, and Run Metadata are attached. | The result cannot explain which candidate configuration was evaluated. | ## Where it appears in the product Use the Benchmark selector to enter a Benchmark workspace. **Benchmark Datasets** owns selected Cases and Snapshots. **Coverage Management** explains representation and gaps. **Expert Contributions** scopes specialist work. **Benchmark Evaluations** runs and compares saved Harness Versions. **Improve** explores evidence-backed candidate changes. ## Artifacts it affects The Benchmark connects Dataset membership, Snapshots, coverage work, Contributions, Benchmark Versions, Runs, comparisons, and Improvement Sessions. Project Cases, Coverage Facets, Policies, Rubrics, and Harnesses remain reusable project objects rather than becoming owned by the Benchmark. {% example-demo title="Coverage refresh version" %} Reviewers discover an unsupported compatibility boundary after a candidate passes Benchmark Version 3. They add reviewed Cases to the current Dataset and create a new Snapshot and Benchmark Version 4. The original Run remains evidence for Version 3. A new Run against Version 4 answers the expanded question, and any customer-owned human review context names both versions. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Run a benchmark](/docs/benchmark-evaluations/run-evaluation) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Read run results](/docs/benchmark-evaluations/inspect-results) - [Work with Benchmark Datasets](/docs/benchmark-datasets) - [Refresh coverage after product change](/docs/coverage-engineering/coverage-refresh) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmarks](/docs/object-model/benchmarks) - [Dataset Snapshots](/docs/benchmark-datasets/snapshots) - [Benchmark Versioning](/docs/governance/benchmark-versioning) - [Reproducibility](/docs/governance/reproducibility) {% /related-card-grid %} ## Source confidence Code-backed: the Benchmark type establishes durable identity; Dataset Snapshots freeze selected Case membership; the version-scoped evaluation workspace consumes an exact Benchmark Version; current navigation establishes the benchmark workspaces named above. --- id: coverage.benchmarks title: Benchmarks summary: Create and manage benchmark sets that measure important AI product behavior. kind: concept product_area: coverage_engineering status: stable updated: 2026-09-07 canonical: /docs/coverage-engineering/benchmarks --- # Benchmarks ## Definition A Benchmark is the durable project object that organizes one intended evaluation program. It owns benchmark-scoped work across Benchmark Datasets, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improve. The Benchmark itself persists while its selected Cases, coverage work, standards, and executable candidates evolve through explicit versions and related artifacts. Do not use “benchmark” interchangeably with a Case collection, a score, or a single Run. The current Benchmark Dataset is editable; a Dataset Snapshot and Benchmark Version freeze an evidence boundary; a Run evaluates an exact saved Harness Version against that boundary. ## Why it matters A Benchmark keeps the work for one behavioral claim connected. Teams can see why Cases were selected, which expert work established the standard, what candidate version ran, and what evidence justified the next improvement. Without that object boundary, coverage, evaluator rules, and candidate results can drift independently while still being summarized under one ambiguous score. ## Where it appears in the product The project-level Benchmarks list creates and routes into individual Benchmark workspaces. Inside a Benchmark, use Benchmark Datasets for selected Cases and Snapshots, Coverage Management for representation and gaps, Expert Contributions for scoped specialist work, Benchmark Evaluations for Runs and comparisons, and Improve for evidence-backed candidate changes. ## Artifacts it affects A Benchmark references reusable project Cases, Coverage Facets, Policies, Rubrics, and Harnesses; it does not take ownership away from those project objects. Benchmark-scoped membership, contributions, Snapshots, Runs, and Improvement Sessions retain their own identities. Deleting or editing a project object must not be described as silently rewriting historical Benchmark Versions. ## Operational check Before interpreting a Benchmark result, confirm the Benchmark purpose, exact Benchmark Version, selected Case population, evaluator boundary, saved Harness Version, and Run completeness. When the benchmark's intended behavior changes, update its coverage and dataset deliberately and create a new evidence boundary instead of treating current mutable state as historical truth. {% example-demo title="One benchmark, two evidence boundaries" %} A support-assistant Benchmark initially covers ordinary return requests. After specialists document an exception for opened safety equipment, Coverage Management identifies the missing boundary and the current dataset gains reviewed Cases and a new Rubric relationship. The Benchmark remains the same program, but the team creates a new Snapshot and Benchmark Version. Comparisons name the version so readers can separate candidate improvement from the expanded correctness boundary. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Benchmarks and versions](/docs/concepts/benchmarks-and-versions) - [Create a benchmark](/docs/coverage-engineering/create-a-benchmark) - [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage) - [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots) - [Product quickstart](/docs/quickstart) {% /related-card-grid %} ## Source confidence Code-backed: the active Benchmarks list, benchmark-scoped workspace routes, and Benchmark types establish the durable Benchmark identity and its relationship to datasets, versions, and evaluation work. Linked pages define the exact behavior of each workspace. --- id: coverage.benchmark-snapshots title: Benchmark snapshots summary: Freeze a benchmark into a version so every run measures the same cases and judgment rules. kind: concept product_area: coverage_engineering status: stable updated: 2026-08-23 canonical: /docs/coverage-engineering/benchmark-snapshots --- # Benchmark snapshots ## Definition A Benchmark Dataset Snapshot is the immutable evidence boundary created from a benchmark's current editable dataset. It records the selected Cases, eligible Policy and Rubric relationships, representation facts, label, content hash, Case count, and creation time used to establish a Benchmark Version. The live dataset can continue changing after the Snapshot exists. Those later changes do not rewrite an earlier Snapshot or any Run that used its Benchmark Version. ## Why it matters Evaluation results are comparable only when readers know which evidence boundary each Run used. Without the Snapshot identity, a score can move because the Harness changed, because Case membership changed, because evaluator relationships changed, or because representation shifted. Freezing the dataset keeps those explanations separate. ## Where it appears in the product Use **Benchmark Datasets → Snapshots** to run readiness checks, create the immutable Snapshot, and inspect its read-only Cases and Representation. Use Coverage Management to improve gaps before snapshotting. Use Benchmark Evaluations after the resulting Benchmark Version is available. ## Artifacts it affects A Snapshot binds selected Case versions, admitted evaluator relationships, and representation facts. It does not freeze the entire project, lock the current dataset, approve weak Cases, or prove that coverage is sufficient. Project Cases, Policies, Rubrics, and Coverage Facets remain reusable objects with their own lifecycle. Create a new Snapshot when a change affects the claim an evaluation will support—for example Case membership, relevant Case content or materials, admitted evaluator relationships, or the represented behavior boundary. A label-only correction that does not change evidence should not be described as new behavioral coverage. ## Operational check Before creating a Snapshot, confirm that readiness has no blockers, Cases are interpretable, required materials are ready, evaluator links are eligible and approved, and Representation supports the intended benchmark claim. After creation, record the Snapshot or Benchmark Version identity in every Run, comparison, and Improvement Session that depends on it. {% example-demo title="Preserving a coverage expansion" %} A source-authority Coverage Story produces eight reviewed Cases covering conflicts between current and superseded documents. The team adds those Cases to the current dataset and confirms their Rubric relationships. Earlier Runs remain attached to the old Benchmark Version. The operator creates a new Snapshot and uses its resulting version for the next evaluation, making the coverage change explicit instead of presenting the score movement as a Harness-only change. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Benchmarks and versions](/docs/concepts/benchmarks-and-versions) - [Create a Dataset Snapshot](/docs/benchmark-datasets/snapshots) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Product quickstart](/docs/quickstart) - [Task index](/docs/operating-manual/task-index) {% /related-card-grid %} ## Source confidence Code-backed: the active Snapshot manager and benchmark-scoped Snapshots route define readiness, immutable Snapshot identity, content hashes, Case counts, and read-only inspection. Coverage sufficiency remains a human interpretation supported by Representation and Coverage Management. --- id: benchmark-evaluations.inspect title: Inspect Evaluation Results summary: Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/inspect-results --- # Inspect Evaluation Results ## Prerequisites - A visible Run or Run Group with output or evaluation progress. - Access to the exact Benchmark and Harness Version evidence. Start with completeness and identity, then move from aggregate signals to the Cases and evaluator failures that support them. ## Steps 1. Open Dashboard and confirm the Benchmark Version, candidate Harness Version, Run Group type, attempt count, and evaluation progress. 2. Read rankings with their metric family and uncertainty. Distinguish average score, passed at least once, and passed every time over observed Runs. Inspect Run counts and coverage; historical group-specific pass@n and pass^n retain their original meanings. 3. Open **List → Runs** to inspect group and Run status, output progress, evaluation progress, metadata, and available resource telemetry. 4. Open **List → Evaluation results** for the Case summary, outcome, failed Policies, failed Rubrics, and evaluated count. 5. Use Arena for pairwise disagreement or Compare for a Harness matrix across Cases, evaluator facts, or Coverage Facets. 6. Classify the next action as candidate work, evaluator clarification, Case correction, coverage work, external-output remapping, or no action. The List results surface is intentionally compact. Do not claim that it exposes full execution trajectories. The **Traces / Spans** segment currently reports a capability fence because the benchmark API does not provide evaluation execution traces. ## Reading incomplete and repeated evidence An aggregate calculated over fewer evaluable Cases can look better while covering less evidence. Record evaluated, incomplete, and missing counts before comparing candidates. For repeated groups, inspect whether the configured number of attempts exists for every candidate and whether one failed attempt changes the metric interpretation. Cost, tokens, and latency help route operational work but are nullable telemetry. Missing capture means unknown, not free or instantaneous execution. > Evaluator authority > > Policy and Rubric results are the correctness evidence admitted by the Benchmark Version. Rankings and telemetry summarize that evidence; they do not create a new standard. {% example-demo title="Example: apparent gain from incomplete evidence" %} Harness B leads the overall table, but List shows that twelve difficult Cases are still unevaluated for B. Arena also reports incomplete pairs. The operator waits for terminal evidence instead of starting Improve from a ranking that covers a smaller Case population. {% /example-demo %} ## Object and state changes Inspection, filtering, and navigation are read-only. Starting Improve, a Contribution, coverage work, or a later Run creates separate durable work while preserving the inspected evidence. ## Success criteria - Identity, completeness, metric family, and uncertainty are explicit. - Important signals resolve to Cases and admitted evaluator outcomes. - The next action targets the responsible artifact or candidate boundary. ## Common failure modes - Reporting rank without the evaluated population. - Inventing execution traces from the unavailable segment. - Starting candidate work when the Case or Rubric is wrong. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Dataset Snapshots](/docs/benchmark-datasets/snapshots) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Missing outputs](/docs/troubleshooting/missing-outputs) {% /related-card-grid %} ## Source confidence Code-backed: Dashboard, List, Run detail, and workspace types establish result summaries, completion, rankings, repeated metrics, telemetry, and the current trace capability fence. --- id: benchmark-evaluations.run title: Run a Benchmark Evaluation summary: Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/run-evaluation --- # Run a Benchmark Evaluation Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready. ## Prerequisites - A Benchmark Version backed by the intended Dataset Snapshot. - Approved eligible Policies and Rubrics. - At least one saved project Harness Version. - Prepared Harness runtime and required secret grants. - A chosen number of Runs for each selected Harness. ## Steps 1. Open **Benchmark Evaluations** for the intended Benchmark Version. 2. Open Evaluation Settings if you need to adjust the machine configuration. 3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone. 4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume. 5. Supply any requested Run Metadata. Keep credentials out of descriptive fields. 6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one. 7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores. 8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable. ## Evidence created The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage. Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete. > No Draft execution > > A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching. {% example-demo title="Example: two candidates, three attempts" %} Harness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort. {% /example-demo %} ## Object and state changes Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort. ## Success criteria - Exact Harness and Benchmark Versions are recorded. - Each launch group contains the number of Runs requested for that Harness. - Output and evaluation progress reach an interpretable terminal state. - Incomplete or failed attempts remain visible. ## Common failure modes - Selecting the wrong saved Version or assuming Draft execution. - Reading partial evaluation as a complete cohort. - Treating absent telemetry as zero usage. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings) - [Harnesses](/docs/assets/harnesses) - [Run Metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Missing outputs](/docs/troubleshooting/missing-outputs) - [Authentication](/docs/troubleshooting/authentication) {% /related-card-grid %} ## Source confidence Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity.