Teammately Docs
Docs menu

concept

Benchmark snapshots

Freeze a benchmark into a version so every run measures the same cases and judgment rules.

Benchmark snapshots

Definition

A Benchmark Dataset Snapshot is the immutable evidence boundary created from a benchmark's current editable dataset. It records the selected Cases, eligible Policy and Rubric relationships, representation facts, label, content hash, Case count, and creation time used to establish a Benchmark Version.

The live dataset can continue changing after the Snapshot exists. Those later changes do not rewrite an earlier Snapshot or any Run that used its Benchmark Version.

Why it matters

Evaluation results are comparable only when readers know which evidence boundary each Run used. Without the Snapshot identity, a score can move because the Harness changed, because Case membership changed, because evaluator relationships changed, or because representation shifted. Freezing the dataset keeps those explanations separate.

Where it appears in the product

Use Benchmark Datasets → Snapshots to run readiness checks, create the immutable Snapshot, and inspect its read-only Cases and Representation. Use Coverage Management to improve gaps before snapshotting. Use Benchmark Evaluations after the resulting Benchmark Version is available.

Artifacts it affects

A Snapshot binds selected Case versions, admitted evaluator relationships, and representation facts. It does not freeze the entire project, lock the current dataset, approve weak Cases, or prove that coverage is sufficient. Project Cases, Policies, Rubrics, and Coverage Facets remain reusable objects with their own lifecycle.

Create a new Snapshot when a change affects the claim an evaluation will support—for example Case membership, relevant Case content or materials, admitted evaluator relationships, or the represented behavior boundary. A label-only correction that does not change evidence should not be described as new behavioral coverage.

Operational check

Before creating a Snapshot, confirm that readiness has no blockers, Cases are interpretable, required materials are ready, evaluator links are eligible and approved, and Representation supports the intended benchmark claim. After creation, record the Snapshot or Benchmark Version identity in every Run, comparison, and Improvement Session that depends on it.

Worked example

Preserving a coverage expansion

A source-authority Coverage Story produces eight reviewed Cases covering conflicts between current and superseded documents. The team adds those Cases to the current dataset and confirms their Rubric relationships. Earlier Runs remain attached to the old Benchmark Version. The operator creates a new Snapshot and uses its resulting version for the next evaluation, making the coverage change explicit instead of presenting the score movement as a Harness-only change.

Related workflows

Concept / Object ModelBenchmarks and versionsUnderstand benchmarks, benchmark versions, snapshots, and why versioned measurement matters.Task / Benchmark DatasetsDataset SnapshotsFreeze Cases, evaluator links, and representation facts as an immutable benchmark evidence boundary.Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.Task / Benchmark EvaluationsInspect Evaluation ResultsTrace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.Quickstart / Learning ManualProduct quickstartConfigure one project foundation, one benchmark slice, one expert contribution, one evaluation, and one evidence-backed improvement.Concept / Operating ManualTask indexRoute correctness work to the current project foundation, benchmark workspace, evaluation, or improvement surface.

Source confidence

Code-backed: the active Snapshot manager and benchmark-scoped Snapshots route define readiness, immutable Snapshot identity, content hashes, Case counts, and read-only inspection. Coverage sufficiency remains a human interpretation supported by Representation and Coverage Management.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Object Model Benchmarks and versions Understand benchmarks, benchmark versions, snapshots, and why versioned measurement matters. Benchmark Evaluations Run a Benchmark Evaluation Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups. Benchmark Evaluations Inspect Evaluation Results Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.
AI context