# Benchmarks

Generated: 2026-09-13T04:33:11.136Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="coverage.benchmarks" slug="coverage-engineering/benchmarks" url="https://teammately.ai/docs/coverage-engineering/benchmarks.md" title="Benchmarks" updated="2026-09-07" status="stable">
---
id: coverage.benchmarks
title: Benchmarks
summary: Create and manage benchmark sets that measure important AI product behavior.
kind: concept
product_area: coverage_engineering
status: stable
updated: 2026-09-07
canonical: /docs/coverage-engineering/benchmarks
---

# Benchmarks

## Definition

A Benchmark is the durable project object that organizes one intended evaluation program. It owns benchmark-scoped work across Benchmark Datasets, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improve. The Benchmark itself persists while its selected Cases, coverage work, standards, and executable candidates evolve through explicit versions and related artifacts.

Do not use “benchmark” interchangeably with a Case collection, a score, or a single Run. The current Benchmark Dataset is editable; a Dataset Snapshot and Benchmark Version freeze an evidence boundary; a Run evaluates an exact saved Harness Version against that boundary.

## Why it matters

A Benchmark keeps the work for one behavioral claim connected. Teams can see why Cases were selected, which expert work established the standard, what candidate version ran, and what evidence justified the next improvement. Without that object boundary, coverage, evaluator rules, and candidate results can drift independently while still being summarized under one ambiguous score.

## Where it appears in the product

The project-level Benchmarks list creates and routes into individual Benchmark workspaces. Inside a Benchmark, use Benchmark Datasets for selected Cases and Snapshots, Coverage Management for representation and gaps, Expert Contributions for scoped specialist work, Benchmark Evaluations for Runs and comparisons, and Improve for evidence-backed candidate changes.

## Artifacts it affects

A Benchmark references reusable project Cases, Coverage Facets, Policies, Rubrics, and Harnesses; it does not take ownership away from those project objects. Benchmark-scoped membership, contributions, Snapshots, Runs, and Improvement Sessions retain their own identities. Deleting or editing a project object must not be described as silently rewriting historical Benchmark Versions.

## Operational check

Before interpreting a Benchmark result, confirm the Benchmark purpose, exact Benchmark Version, selected Case population, evaluator boundary, saved Harness Version, and Run completeness. When the benchmark's intended behavior changes, update its coverage and dataset deliberately and create a new evidence boundary instead of treating current mutable state as historical truth.

{% example-demo title="One benchmark, two evidence boundaries" %}
A support-assistant Benchmark initially covers ordinary return requests. After specialists document an exception for opened safety equipment, Coverage Management identifies the missing boundary and the current dataset gains reviewed Cases and a new Rubric relationship. The Benchmark remains the same program, but the team creates a new Snapshot and Benchmark Version. Comparisons name the version so readers can separate candidate improvement from the expanded correctness boundary.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Create a benchmark](/docs/coverage-engineering/create-a-benchmark)
- [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage)
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Product quickstart](/docs/quickstart)
{% /related-card-grid %}

## Source confidence

Code-backed: the active Benchmarks list, benchmark-scoped workspace routes, and Benchmark types establish the durable Benchmark identity and its relationship to datasets, versions, and evaluation work. Linked pages define the exact behavior of each workspace.
</doc>

<doc id="concepts.benchmarks-versions" slug="concepts/benchmarks-and-versions" url="https://teammately.ai/docs/concepts/benchmarks-and-versions.md" title="Benchmarks and versions" updated="2026-08-23" status="stable">
---
id: concepts.benchmarks-versions
title: Benchmarks and versions
summary: Understand benchmarks, benchmark versions, snapshots, and why versioned measurement matters.
kind: concept
product_area: object_model
status: stable
updated: 2026-08-23
canonical: /docs/concepts/benchmarks-and-versions
---

# Benchmarks and versions

## Definition

A Benchmark is the durable workspace for one evaluation program. Its current Dataset and coverage work can change over time. A Dataset Snapshot freezes selected Case membership; a Benchmark Version fixes the evidence boundary used by an Evaluation Run.

## Why it matters

This separation keeps evidence interpretable. A score can change because the saved Harness Version changed, the Dataset changed, or the governed evaluator boundary changed. Naming the Benchmark Version prevents those causes from being collapsed into “the model changed.”

## Version sanity check

| Before comparing runs, confirm... | Otherwise... |
| --- | --- |
| The same benchmark version was used, or the version change is intentional and named. | Score movement may be caused by changed cases or standards. |
| The Dataset Snapshot and governed evaluator boundary are identifiable. | A comparison can hide the evidence that actually changed. |
| The saved Harness Version, settings, and Run Metadata are attached. | The result cannot explain which candidate configuration was evaluated. |

## Where it appears in the product

Use the Benchmark selector to enter a Benchmark workspace. **Benchmark Datasets** owns selected Cases and Snapshots. **Coverage Management** explains representation and gaps. **Expert Contributions** scopes specialist work. **Benchmark Evaluations** runs and compares saved Harness Versions. **Improve** explores evidence-backed candidate changes.

## Artifacts it affects

The Benchmark connects Dataset membership, Snapshots, coverage work, Contributions, Benchmark Versions, Runs, comparisons, and Improvement Sessions. Project Cases, Coverage Facets, Policies, Rubrics, and Harnesses remain reusable project objects rather than becoming owned by the Benchmark.

{% example-demo title="Coverage refresh version" %}
Reviewers discover an unsupported compatibility boundary after a candidate passes Benchmark Version 3. They add reviewed Cases to the current Dataset and create a new Snapshot and Benchmark Version 4. The original Run remains evidence for Version 3. A new Run against Version 4 answers the expanded question, and any customer-owned human review context names both versions.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Run a benchmark](/docs/benchmark-evaluations/run-evaluation)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Read run results](/docs/benchmark-evaluations/inspect-results)
- [Work with Benchmark Datasets](/docs/benchmark-datasets)
- [Refresh coverage after product change](/docs/coverage-engineering/coverage-refresh)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmarks](/docs/object-model/benchmarks)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
- [Benchmark Versioning](/docs/governance/benchmark-versioning)
- [Reproducibility](/docs/governance/reproducibility)
{% /related-card-grid %}

## Source confidence

Code-backed: the Benchmark type establishes durable identity; Dataset Snapshots freeze selected Case membership; the version-scoped evaluation workspace consumes an exact Benchmark Version; current navigation establishes the benchmark workspaces named above.
</doc>

<doc id="coverage.create-benchmark" slug="coverage-engineering/create-a-benchmark" url="https://teammately.ai/docs/coverage-engineering/create-a-benchmark.md" title="Create a benchmark" updated="2026-08-23" status="stable">
---
id: coverage.create-benchmark
title: Create a benchmark
summary: Create the durable Benchmark workspace in which you will define coverage, select Cases, and create reproducible Snapshots.
kind: task
product_area: coverage_engineering
status: stable
updated: 2026-08-23
canonical: /docs/coverage-engineering/create-a-benchmark
---

# Create a benchmark

## Prerequisites

Write one sentence describing the behavior claim the Benchmark should support. You should also know the target system, the intended reviewers, and where candidate Cases will come from. You do not need a finished Case set to create the Benchmark.

### Task steps: Create and prepare a benchmark

1. Open the Benchmark selector in the project navigation.
2. Select **Create New Benchmark**.
3. Enter a name that identifies the target behavior or decision boundary, then select **Create Benchmark**.
4. Open the new Benchmark and describe its purpose before curating evidence.
5. Open **Coverage Management → Get Started** to define the benchmark denominator and coverage guidance.
6. Use **Coverage Management** and **Benchmark Datasets** to prepare, review, and select Cases. Keep candidate material distinct from the selected Dataset.
7. Create a Dataset Snapshot only when the selected Cases are ready to become an immutable evidence boundary.

## Object and state changes

Creating a Benchmark establishes its identity and workspace; it does not create a complete Benchmark Version. The initial description is empty in the current creation flow, and the default name is **New Benchmark** when no name is supplied. Rename and describe it before teammates depend on it.

A Benchmark can evolve through coverage planning and Case selection. A Snapshot is the point at which a particular evidence set becomes reproducible. A Run belongs to a Benchmark Version; it is not the Benchmark itself.

## Name benchmarks for durable interpretation

Prefer a name such as **Support assistant — refund eligibility** over **August test**. Dates and change markers belong in Snapshots, Benchmark Versions, or version notes. The Benchmark name should remain meaningful as the Case set improves.

## Success criteria

- The Benchmark has a durable name and an explicit behavior claim.
- Its owner can explain the target system and intended decision.
- Coverage planning identifies what must be represented before Snapshot creation.
- Candidate Cases are not treated as selected benchmark evidence by default.

## Common failure modes

- Naming the Benchmark after a date or experiment rather than its durable behavior claim.
- Treating creation as though a complete Benchmark Version or Snapshot now exists.
- Selecting convenient Cases before defining the intended coverage boundary.
- Starting Runs before selected Cases and governed standards are ready.

{% example-demo title="Support escalation benchmark" %}
An AI engineer creates **Support assistant — escalation decisions**. The description says the Benchmark tests whether the assistant escalates high-risk cases while resolving routine ones. The team plans risk, account tier, and source-authority coverage, then curates Cases from the Case Pool. Only after review does the team create its first Snapshot.
{% /example-demo %}

## Source confidence

Code-backed: the Benchmark selector and creation modal define the current creation path; Coverage Management Get Started and Benchmark Datasets define the immediate next work. A newly created Benchmark is a durable workspace, not ready evaluation evidence.

## Related reference pages

{% related-card-grid title="Continue the workflow" %}
- [Benchmarks](/docs/coverage-engineering/benchmarks)
- [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage)
- [Case Pool](/docs/coverage-engineering/case-pool)
- [Benchmark Snapshots](/docs/coverage-engineering/benchmark-snapshots)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Diagnose benchmark setup" %}
- [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage)
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
- [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs)
{% /related-card-grid %}
</doc>

<doc id="coverage.plan-benchmark-coverage" slug="coverage-engineering/plan-benchmark-coverage" url="https://teammately.ai/docs/coverage-engineering/plan-benchmark-coverage.md" title="Plan Benchmark Coverage" updated="2026-09-07" status="stable">
---
id: coverage.plan-benchmark-coverage
title: Plan Benchmark Coverage
summary: Apply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work.
kind: task
product_area: coverage_engineering
status: stable
updated: 2026-09-07
canonical: /docs/coverage-engineering/plan-benchmark-coverage
---

# Plan Benchmark Coverage

Plan coverage by applying reusable project facets to one benchmark and comparing the intended behavior space with the selected dataset representation.

## Prerequisites

- A selected project and benchmark.
- A clear benchmark purpose.
- Relevant Dimensions, Project Topics, and Case Construction Patterns, or enough project knowledge to create them.
- Existing Cases or a plan for sourcing and constructing them.

## Steps

1. Review **Coverage Facets** at project scope. Confirm that Dimensions, Project Topics, and Case Construction Patterns describe reusable behavior structure rather than one benchmark's current case count.
2. Open the benchmark and select **Coverage Management → Get Started**.
3. Define the benchmark-specific coverage guidance and confirm setup readiness.
4. Open Coverage Management and inspect current dataset representation across the relevant facets and tuples.
5. Name important thin or absent combinations as Coverage Stories. Explain why each slice matters and what evidence would make it usable.
6. Route the gap according to its cause: Case Foundry or case sourcing for missing situations, Expert Contributions for missing judgment, Correctness Governance for missing standards, or Benchmark Datasets for missing selection.
7. Review generated or contributed cases in Case Review before relying on them.
8. Update dataset selection and create a new snapshot when the represented evidence changes materially.

## Object and state changes

This task can update benchmark coverage setup, representation guidance, Coverage Stories, Case Foundry work, case-review state, contribution requests, dataset selection, and snapshots. Project Coverage Facets may also change when the work discovers a reusable missing axis or construction pattern.

## Success criteria

- The benchmark purpose maps to explicit project Coverage Facets.
- Important combinations have selected evidence or a named gap.
- Each gap is routed to a responsible artifact or workstream.
- Constructed cases pass case review and Project Input Schema checks.
- Dataset snapshots make material coverage changes explicit.

## Common failure modes

- Using case count as the coverage goal.
- Creating benchmark-only tags where a reusable Dimension or Topic is needed.
- Treating response-variation guidance as coverage structure.
- Generating cases before defining which gap they should close.
- Trusting representation after selection changes without a new snapshot boundary.

{% example-demo title="Example: plan high-impact exception coverage" %}
The team maps exception type, source authority, and customer impact. Representation shows many low-impact ordinary cases but no high-impact cases with conflicting authority. A Coverage Story names the gap, an expert Contribution clarifies the controlling rule, and Case Foundry prepares cases for the missing tuple before a new snapshot is created.
{% /example-demo %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Coverage Engineering](/docs/coverage-engineering)
- [Coverage Management](/docs/coverage-management)
- [Benchmark Datasets](/docs/benchmark-datasets)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage)
- [Stale dimensions](/docs/troubleshooting/stale-dimensions)
- [Unrealistic synthetic cases](/docs/troubleshooting/unrealistic-synthetic-cases)
{% /related-card-grid %}

## Source confidence

Code-backed: current setup, overview, representation, Coverage Story, Case Foundry, and Case Review routes support this workflow.
</doc>

<doc id="coverage.benchmark-snapshots" slug="coverage-engineering/benchmark-snapshots" url="https://teammately.ai/docs/coverage-engineering/benchmark-snapshots.md" title="Benchmark snapshots" updated="2026-08-23" status="stable">
---
id: coverage.benchmark-snapshots
title: Benchmark snapshots
summary: Freeze a benchmark into a version so every run measures the same cases and judgment rules.
kind: concept
product_area: coverage_engineering
status: stable
updated: 2026-08-23
canonical: /docs/coverage-engineering/benchmark-snapshots
---

# Benchmark snapshots

## Definition

A Benchmark Dataset Snapshot is the immutable evidence boundary created from a benchmark's current editable dataset. It records the selected Cases, eligible Policy and Rubric relationships, representation facts, label, content hash, Case count, and creation time used to establish a Benchmark Version.

The live dataset can continue changing after the Snapshot exists. Those later changes do not rewrite an earlier Snapshot or any Run that used its Benchmark Version.

## Why it matters

Evaluation results are comparable only when readers know which evidence boundary each Run used. Without the Snapshot identity, a score can move because the Harness changed, because Case membership changed, because evaluator relationships changed, or because representation shifted. Freezing the dataset keeps those explanations separate.

## Where it appears in the product

Use **Benchmark Datasets → Snapshots** to run readiness checks, create the immutable Snapshot, and inspect its read-only Cases and Representation. Use Coverage Management to improve gaps before snapshotting. Use Benchmark Evaluations after the resulting Benchmark Version is available.

## Artifacts it affects

A Snapshot binds selected Case versions, admitted evaluator relationships, and representation facts. It does not freeze the entire project, lock the current dataset, approve weak Cases, or prove that coverage is sufficient. Project Cases, Policies, Rubrics, and Coverage Facets remain reusable objects with their own lifecycle.

Create a new Snapshot when a change affects the claim an evaluation will support—for example Case membership, relevant Case content or materials, admitted evaluator relationships, or the represented behavior boundary. A label-only correction that does not change evidence should not be described as new behavioral coverage.

## Operational check

Before creating a Snapshot, confirm that readiness has no blockers, Cases are interpretable, required materials are ready, evaluator links are eligible and approved, and Representation supports the intended benchmark claim. After creation, record the Snapshot or Benchmark Version identity in every Run, comparison, and Improvement Session that depends on it.

{% example-demo title="Preserving a coverage expansion" %}
A source-authority Coverage Story produces eight reviewed Cases covering conflicts between current and superseded documents. The team adds those Cases to the current dataset and confirms their Rubric relationships. Earlier Runs remain attached to the old Benchmark Version. The operator creates a new Snapshot and uses its resulting version for the next evaluation, making the coverage change explicit instead of presenting the score movement as a Harness-only change.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Create a Dataset Snapshot](/docs/benchmark-datasets/snapshots)
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
- [Product quickstart](/docs/quickstart)
- [Task index](/docs/operating-manual/task-index)
{% /related-card-grid %}

## Source confidence

Code-backed: the active Snapshot manager and benchmark-scoped Snapshots route define readiness, immutable Snapshot identity, content hashes, Case counts, and read-only inspection. Coverage sufficiency remains a human interpretation supported by Representation and Coverage Management.
</doc>