---
id: coverage.benchmarks
title: Benchmarks
summary: Create and manage benchmark sets that measure important AI product behavior.
kind: concept
product_area: coverage_engineering
status: stable
updated: 2026-09-07
canonical: /docs/coverage-engineering/benchmarks
---

# Benchmarks

## Definition

A Benchmark is the durable project object that organizes one intended evaluation program. It owns benchmark-scoped work across Benchmark Datasets, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improve. The Benchmark itself persists while its selected Cases, coverage work, standards, and executable candidates evolve through explicit versions and related artifacts.

Do not use “benchmark” interchangeably with a Case collection, a score, or a single Run. The current Benchmark Dataset is editable; a Dataset Snapshot and Benchmark Version freeze an evidence boundary; a Run evaluates an exact saved Harness Version against that boundary.

## Why it matters

A Benchmark keeps the work for one behavioral claim connected. Teams can see why Cases were selected, which expert work established the standard, what candidate version ran, and what evidence justified the next improvement. Without that object boundary, coverage, evaluator rules, and candidate results can drift independently while still being summarized under one ambiguous score.

## Where it appears in the product

The project-level Benchmarks list creates and routes into individual Benchmark workspaces. Inside a Benchmark, use Benchmark Datasets for selected Cases and Snapshots, Coverage Management for representation and gaps, Expert Contributions for scoped specialist work, Benchmark Evaluations for Runs and comparisons, and Improve for evidence-backed candidate changes.

## Artifacts it affects

A Benchmark references reusable project Cases, Coverage Facets, Policies, Rubrics, and Harnesses; it does not take ownership away from those project objects. Benchmark-scoped membership, contributions, Snapshots, Runs, and Improvement Sessions retain their own identities. Deleting or editing a project object must not be described as silently rewriting historical Benchmark Versions.

## Operational check

Before interpreting a Benchmark result, confirm the Benchmark purpose, exact Benchmark Version, selected Case population, evaluator boundary, saved Harness Version, and Run completeness. When the benchmark's intended behavior changes, update its coverage and dataset deliberately and create a new evidence boundary instead of treating current mutable state as historical truth.

{% example-demo title="One benchmark, two evidence boundaries" %}
A support-assistant Benchmark initially covers ordinary return requests. After specialists document an exception for opened safety equipment, Coverage Management identifies the missing boundary and the current dataset gains reviewed Cases and a new Rubric relationship. The Benchmark remains the same program, but the team creates a new Snapshot and Benchmark Version. Comparisons name the version so readers can separate candidate improvement from the expanded correctness boundary.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Create a benchmark](/docs/coverage-engineering/create-a-benchmark)
- [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage)
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Product quickstart](/docs/quickstart)
{% /related-card-grid %}

## Source confidence

Code-backed: the active Benchmarks list, benchmark-scoped workspace routes, and Benchmark types establish the durable Benchmark identity and its relationship to datasets, versions, and evaluation work. Linked pages define the exact behavior of each workspace.