---
id: concepts.benchmarks-versions
title: Benchmarks and versions
summary: Understand benchmarks, benchmark versions, snapshots, and why versioned measurement matters.
kind: concept
product_area: object_model
status: stable
updated: 2026-08-23
canonical: /docs/concepts/benchmarks-and-versions
---

# Benchmarks and versions

## Definition

A Benchmark is the durable workspace for one evaluation program. Its current Dataset and coverage work can change over time. A Dataset Snapshot freezes selected Case membership; a Benchmark Version fixes the evidence boundary used by an Evaluation Run.

## Why it matters

This separation keeps evidence interpretable. A score can change because the saved Harness Version changed, the Dataset changed, or the governed evaluator boundary changed. Naming the Benchmark Version prevents those causes from being collapsed into “the model changed.”

## Version sanity check

| Before comparing runs, confirm... | Otherwise... |
| --- | --- |
| The same benchmark version was used, or the version change is intentional and named. | Score movement may be caused by changed cases or standards. |
| The Dataset Snapshot and governed evaluator boundary are identifiable. | A comparison can hide the evidence that actually changed. |
| The saved Harness Version, settings, and Run Metadata are attached. | The result cannot explain which candidate configuration was evaluated. |

## Where it appears in the product

Use the Benchmark selector to enter a Benchmark workspace. **Benchmark Datasets** owns selected Cases and Snapshots. **Coverage Management** explains representation and gaps. **Expert Contributions** scopes specialist work. **Benchmark Evaluations** runs and compares saved Harness Versions. **Improve** explores evidence-backed candidate changes.

## Artifacts it affects

The Benchmark connects Dataset membership, Snapshots, coverage work, Contributions, Benchmark Versions, Runs, comparisons, and Improvement Sessions. Project Cases, Coverage Facets, Policies, Rubrics, and Harnesses remain reusable project objects rather than becoming owned by the Benchmark.

{% example-demo title="Coverage refresh version" %}
Reviewers discover an unsupported compatibility boundary after a candidate passes Benchmark Version 3. They add reviewed Cases to the current Dataset and create a new Snapshot and Benchmark Version 4. The original Run remains evidence for Version 3. A new Run against Version 4 answers the expanded question, and any customer-owned human review context names both versions.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Run a benchmark](/docs/benchmark-evaluations/run-evaluation)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Read run results](/docs/benchmark-evaluations/inspect-results)
- [Work with Benchmark Datasets](/docs/benchmark-datasets)
- [Refresh coverage after product change](/docs/coverage-engineering/coverage-refresh)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmarks](/docs/object-model/benchmarks)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
- [Benchmark Versioning](/docs/governance/benchmark-versioning)
- [Reproducibility](/docs/governance/reproducibility)
{% /related-card-grid %}

## Source confidence

Code-backed: the Benchmark type establishes durable identity; Dataset Snapshots freeze selected Case membership; the version-scoped evaluation workspace consumes an exact Benchmark Version; current navigation establishes the benchmark workspaces named above.