Teammately Docs
Docs menu

reference

Benchmarks

Understand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.

Benchmarks

Definition

A Benchmark is the durable project object for one intended evaluation program. It owns benchmark-scoped work across Benchmark Datasets, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improve while its selected Cases, governed standards, and candidate systems evolve.

The current Benchmark Dataset is editable. A Dataset Snapshot freezes selected Case membership, and a Benchmark Version provides the immutable boundary consumed by Runs. A Benchmark is therefore not a Snapshot, Benchmark Version, Run, or score.

Fields, states, or lifecycle rules

  • The Benchmark identity persists across changes to its current Dataset, coverage work, standards, and Harness candidates.
  • Benchmark Datasets owns selected Cases and immutable Dataset Snapshots.
  • A Benchmark Version fixes the evidence boundary used by a Run.
  • Benchmark membership should be shaped by coverage work, not by whichever Cases are easiest to run.
  • A Run result is weak if the Benchmark Version and saved Harness Version behind it are unclear.
  • This page documents object semantics, not public execution, export, rate-limit, or API guarantees.

Benchmarks should be read with Cases, Policies, Rubrics, Coverage Engineering, and Benchmark Evaluations. Use Run an evaluation when the next step is execution.

Worked example

Benchmarks boundary

01

Start

Behavior input

Raw case
A team refreshes coverage after finding unsupported compatibility claims.
02

Middle

Judgment into standard

Benchmark version
The refreshed version includes new unsupported-claim cases and the approved compatibility rubric.
Run
The candidate is evaluated against that version.
03

Result

Interpretation

Interpretation
If the score drops, reviewers can see that the benchmark became harder instead of assuming the candidate behavior changed.

Source confidence

Code-backed: the Benchmark type and workspace establish durable Benchmark identity; Benchmark Datasets → Snapshots establishes immutable Dataset boundaries; the evaluation-runs route consumes a specific Benchmark Version. The public object definition does not imply an execution or export API.

Related task pages

Concept / Coverage EngineeringBenchmarksCreate and manage benchmark sets that measure important AI product behavior.Concept / Object ModelBenchmarks and versionsUnderstand benchmarks, benchmark versions, snapshots, and why versioned measurement matters.Concept / Benchmark EvaluationsBenchmark EvaluationsRun and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare.Quickstart / Learning ManualProduct quickstartConfigure one project foundation, one benchmark slice, one expert contribution, one evaluation, and one evidence-backed improvement.Concept / Operating ManualTask indexRoute correctness work to the current project foundation, benchmark workspace, evaluation, or improvement surface.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Benchmarks Create and manage benchmark sets that measure important AI product behavior. Object Model Benchmarks and versions Understand benchmarks, benchmark versions, snapshots, and why versioned measurement matters. Benchmark Datasets Dataset Snapshots Freeze Cases, evaluator links, and representation facts as an immutable benchmark evidence boundary.
AI context