Teammately Docs
Docs menu

reference

Benchmark Versioning

Preserve benchmark snapshots so evidence can be compared across target and standard changes.

Benchmark Versioning

Definition

A Benchmark Version is the immutable evidence boundary used by Runs. It identifies the frozen dataset state and admitted evaluator relationships that make a result interpretable. The Benchmark remains a durable program; its versions preserve successive evidence boundaries as Cases, materials, coverage, Policies, or Rubrics change.

Fields, states, or lifecycle rules

  • Create a new Snapshot and resulting Benchmark Version when changed evidence would alter what a Run claims to test.
  • Existing Runs remain attached to their original Benchmark Version.
  • Current Dataset edits do not mutate a historical version.
  • A new Harness Version alone does not require a new Benchmark Version; candidate and evidence versions move independently.
  • Comparisons within one Benchmark Version isolate candidate differences more cleanly.
  • Cross-version comparisons must name the changed Cases, evaluators, or representation boundary.
  • Version identity does not prove that coverage is sufficient or that every admitted Rubric is correct.

Use Dataset Snapshots to create the frozen dataset boundary. Use Benchmark Evaluations to inspect Runs inside one exact version, and Compare Harness Versions to interpret candidate movement without hiding version changes.

Worked example

Separating candidate change from standard change

Harness Version 12 improves retrieval and is evaluated against Benchmark Version 4, the same boundary used for Version 11. That comparison isolates candidate behavior. Later, experts approve a stricter source-authority Rubric and the dataset gains conflict Cases. The team creates Benchmark Version 5 and reports subsequent Runs under that new boundary instead of presenting the lower score as a regression against unchanged evidence.

Source confidence

Code-backed: Benchmark Datasets → Snapshots preserves immutable Dataset boundaries, and the version-scoped Evaluation route binds Runs to one selected Benchmark Version. Coverage quality and downstream decisions remain outside version identity itself.

Related task pages

Concept / Coverage EngineeringBenchmark snapshotsFreeze a benchmark into a version so every run measures the same cases and judgment rules.Reference / Object ModelBenchmarksUnderstand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.Task / Benchmark EvaluationsCompare Harness VersionsCompare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.Quickstart / Learning ManualProduct quickstartConfigure one project foundation, one benchmark slice, one expert contribution, one evaluation, and one evidence-backed improvement.Concept / Operating ManualTask indexRoute correctness work to the current project foundation, benchmark workspace, evaluation, or improvement surface.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Benchmark snapshots Freeze a benchmark into a version so every run measures the same cases and judgment rules. Object Model Benchmarks Understand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work. Benchmark Evaluations Compare Harness Versions Compare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.
AI context