Teammately Docs
Docs menu

concept

Benchmarks and versions

Understand benchmarks, benchmark versions, snapshots, and why versioned measurement matters.

Benchmarks and versions

Definition

A Benchmark is the durable workspace for one evaluation program. Its current Dataset and coverage work can change over time. A Dataset Snapshot freezes selected Case membership; a Benchmark Version fixes the evidence boundary used by an Evaluation Run.

Why it matters

This separation keeps evidence interpretable. A score can change because the saved Harness Version changed, the Dataset changed, or the governed evaluator boundary changed. Naming the Benchmark Version prevents those causes from being collapsed into “the model changed.”

Version sanity check

Before comparing runs, confirm...Otherwise...
The same benchmark version was used, or the version change is intentional and named.Score movement may be caused by changed cases or standards.
The Dataset Snapshot and governed evaluator boundary are identifiable.A comparison can hide the evidence that actually changed.
The saved Harness Version, settings, and Run Metadata are attached.The result cannot explain which candidate configuration was evaluated.

Where it appears in the product

Use the Benchmark selector to enter a Benchmark workspace. Benchmark Datasets owns selected Cases and Snapshots. Coverage Management explains representation and gaps. Expert Contributions scopes specialist work. Benchmark Evaluations runs and compares saved Harness Versions. Improve explores evidence-backed candidate changes.

Artifacts it affects

The Benchmark connects Dataset membership, Snapshots, coverage work, Contributions, Benchmark Versions, Runs, comparisons, and Improvement Sessions. Project Cases, Coverage Facets, Policies, Rubrics, and Harnesses remain reusable project objects rather than becoming owned by the Benchmark.

Worked example

Coverage refresh version

Reviewers discover an unsupported compatibility boundary after a candidate passes Benchmark Version 3. They add reviewed Cases to the current Dataset and create a new Snapshot and Benchmark Version 4. The original Run remains evidence for Version 3. A new Run against Version 4 answers the expanded question, and any customer-owned human review context names both versions.

Related workflows

Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.Task / Benchmark EvaluationsCompare Harness VersionsCompare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.Task / Benchmark EvaluationsInspect Evaluation ResultsTrace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.Concept / Benchmark DatasetsBenchmark DatasetsSelect benchmark Cases, inspect representation, and freeze immutable Snapshots for reproducible evidence.Task / Coverage EngineeringRefresh coverage after product changeReconcile coverage facets, Cases, benchmark membership, and Snapshots after the target system or its evidence changes.

Related reference pages

Reference / Object ModelBenchmarksUnderstand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.Task / Benchmark DatasetsDataset SnapshotsFreeze Cases, evaluator links, and representation facts as an immutable benchmark evidence boundary.Reference / Governance and ReproducibilityBenchmark VersioningPreserve benchmark snapshots so evidence can be compared across target and standard changes.Reference / Governance and ReproducibilityReproducibilityPreserve enough source context to explain and repeat correctness decisions.

Source confidence

Code-backed: the Benchmark type establishes durable identity; Dataset Snapshots freeze selected Case membership; the version-scoped evaluation workspace consumes an exact Benchmark Version; current navigation establishes the benchmark workspaces named above.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Benchmarks Create and manage benchmark sets that measure important AI product behavior. Coverage Engineering Benchmark snapshots Freeze a benchmark into a version so every run measures the same cases and judgment rules. Benchmark Evaluations Inspect Evaluation Results Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence. Benchmark Evaluations Run a Benchmark Evaluation Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.
AI context