# Reproducibility
Generated: 2026-09-13T04:39:37.539Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: governance.reproducibility
title: Reproducibility
summary: Preserve enough source context to explain and repeat correctness decisions.
kind: reference
product_area: governance
status: stable
updated: 2026-08-23
canonical: /docs/governance/reproducibility
---
# Reproducibility
## Definition
Reproducibility means preserving enough exact identity and observable configuration to explain what was evaluated and to repeat the supported execution path. It does not mean every future execution will produce an identical stochastic output. It means a reader can distinguish changes in candidate, evidence, evaluator, sampling, and runtime metadata instead of attributing every result difference to the model.
## Fields, states, or lifecycle rules
- Preserve Project, Benchmark, Benchmark Version, Dataset Snapshot, and Run identity.
- Preserve the exact saved Harness Version rather than an editable draft or display label.
- Preserve admitted Case, Policy, and Rubric version boundaries through the Benchmark Version.
- Record Run Group, attempt, sampling profile, evaluator set, and visible execution settings.
- Retain run metadata and measured telemetry when captured; missing values remain unknown.
- Record completeness, incomplete Cases, and terminal state beside scores.
- Use canonical evaluation receipts for Improvement Session candidate claims.
- Do not claim private worker activity, hidden reasoning, infrastructure internals, or unavailable traces as reproducibility evidence.
## Related objects
Dataset Snapshots preserve the evidence set. Benchmark Evaluations preserves candidate, Run, settings, results, and available telemetry. Compare and Arena interpret candidates inside compatible evidence boundaries. Improve adds Goal Contract, candidate, and canonical receipt identity when evaluation drives code or Harness changes.
{% example-demo title="Diagnosing a score change" %}
Two Runs use the same Harness Version but report different pass rates. The operator confirms that one Run used Benchmark Version 6 and the other used Version 7, which added source-conflict Cases and a revised grounding Rubric. The version and completeness record explains the movement. The team avoids filing a candidate regression until it compares Runs inside the same evidence boundary.
{% /example-demo %}
## Source confidence
Code-backed: Snapshot, Run detail, and run-metadata surfaces expose the immutable evidence boundary, candidate identity, status, counts, settings, and available metadata needed for supported reproducibility. They do not promise deterministic model output or unrestricted execution traces.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Benchmark Versioning](/docs/governance/benchmark-versioning)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Product quickstart](/docs/quickstart)
- [Task index](/docs/operating-manual/task-index)
{% /related-card-grid %}
---
id: governance.benchmark-versioning
title: Benchmark Versioning
summary: Preserve benchmark snapshots so evidence can be compared across target and standard changes.
kind: reference
product_area: governance
status: stable
updated: 2026-08-23
canonical: /docs/governance/benchmark-versioning
---
# Benchmark Versioning
## Definition
A Benchmark Version is the immutable evidence boundary used by Runs. It identifies the frozen dataset state and admitted evaluator relationships that make a result interpretable. The Benchmark remains a durable program; its versions preserve successive evidence boundaries as Cases, materials, coverage, Policies, or Rubrics change.
## Fields, states, or lifecycle rules
- Create a new Snapshot and resulting Benchmark Version when changed evidence would alter what a Run claims to test.
- Existing Runs remain attached to their original Benchmark Version.
- Current Dataset edits do not mutate a historical version.
- A new Harness Version alone does not require a new Benchmark Version; candidate and evidence versions move independently.
- Comparisons within one Benchmark Version isolate candidate differences more cleanly.
- Cross-version comparisons must name the changed Cases, evaluators, or representation boundary.
- Version identity does not prove that coverage is sufficient or that every admitted Rubric is correct.
## Related objects
Use [Dataset Snapshots](/docs/benchmark-datasets/snapshots) to create the frozen dataset boundary. Use [Benchmark Evaluations](/docs/benchmark-evaluations) to inspect Runs inside one exact version, and [Compare Harness Versions](/docs/benchmark-evaluations/compare) to interpret candidate movement without hiding version changes.
{% example-demo title="Separating candidate change from standard change" %}
Harness Version 12 improves retrieval and is evaluated against Benchmark Version 4, the same boundary used for Version 11. That comparison isolates candidate behavior. Later, experts approve a stricter source-authority Rubric and the dataset gains conflict Cases. The team creates Benchmark Version 5 and reports subsequent Runs under that new boundary instead of presenting the lower score as a regression against unchanged evidence.
{% /example-demo %}
## Source confidence
Code-backed: Benchmark Datasets → Snapshots preserves immutable Dataset boundaries, and the version-scoped Evaluation route binds Runs to one selected Benchmark Version. Coverage quality and downstream decisions remain outside version identity itself.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Benchmarks](/docs/object-model/benchmarks)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Product quickstart](/docs/quickstart)
- [Task index](/docs/operating-manual/task-index)
{% /related-card-grid %}
---
id: benchmark-evaluations.compare
title: Compare Harness Versions
summary: Compare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/compare
---
# Compare Harness Versions
Compare shows aggregate observed evidence for saved Harness Versions. Choose the Versions, compatible evaluation configuration, and measurement to compare. Use **Inspect individual Run comparisons** for the detailed result matrix.
## Prerequisites
Saved Harness Versions with evaluation results are shown for the selected Benchmark Version. Choose one or more Versions to inspect; Run counts may differ. An output-only imported reference Run cannot become a Harness column because it has no executable saved Version.
## Select a row mode
The individual Run matrix offers row modes including Cases, all results, Policies, Rubrics, Dimension ontology values, Project Topics, Topic Groups, and Case Construction Patterns. Use Cases to inspect concrete disagreement, Policies or Rubrics to locate correctness movement, and Coverage Facets to see whether gains concentrate in one behavior slice.
## Steps
1. Confirm the immutable Benchmark Version and choose at least two visible Harness Versions.
2. Select an aggregate measurement, or open individual Run comparisons and choose a row mode.
3. Check evidence completeness for each Harness column. A blank or incomplete cell is not a failure.
4. Locate rows with material disagreement and connect them back to Case and evaluator evidence.
5. Preserve regressions and required-criterion failures next to gains.
6. Use the exact candidate and row evidence when starting an Improvement Session or requesting an Expert Contribution.
Compare reads existing evidence and does not mutate Runs. Selecting aggregate Harness Versions recomputes their comparison on compatible evidence; individual matrix visibility is local presentation. It does not activate Harnesses or choose a winner.
{% example-demo title="Example: facet-local improvement" %}
Three Harness Versions look similar overall. The Project Topic row mode shows that Version 14 improves source-authority Topics but regresses escalation Topics. Switching to Cases identifies two regressions, and the team starts Improve with those exact failures instead of claiming a uniform improvement.
{% /example-demo %}
## Common mistakes
- Comparing different Benchmark Versions as though only the candidate moved.
- Treating missing evidence as a failed cell.
- Reading a facet aggregate without checking the distinct Cases behind it.
- Describing an imported output-only Run as a Harness Version.
- Selecting the newest Version solely because it is newest.
## Object and state changes
Compare reads existing evidence. Selecting aggregate Versions reads their observed evidence; selecting individual matrix row modes changes presentation; it does not activate a Harness, mutate a Run, or retain a candidate.
## Success criteria
- At least two exact Harness Versions share the same Benchmark Version.
- Incomplete cells remain distinct from failed evidence.
- Material movement resolves to Cases, evaluators, or Coverage Facets.
## Common failure modes
- Comparing moved evidence boundaries as candidate-only change.
- Treating local column visibility as product configuration.
- Using an output-only Run as a Harness column.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage)
{% /related-card-grid %}
## Source confidence
Code-backed: the active Compare route and Evaluation Matrix define Harness columns, local visibility, symmetric comparison, and the current evidence row modes.
---
id: object-model.benchmarks
title: Benchmarks
summary: Understand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.
kind: reference
product_area: object_model
status: stable
updated: 2026-09-07
canonical: /docs/object-model/benchmarks
---
# Benchmarks
## Definition
A Benchmark is the durable project object for one intended evaluation program. It owns benchmark-scoped work across Benchmark Datasets, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improve while its selected Cases, governed standards, and candidate systems evolve.
The current Benchmark Dataset is editable. A Dataset Snapshot freezes selected Case membership, and a Benchmark Version provides the immutable boundary consumed by Runs. A Benchmark is therefore not a Snapshot, Benchmark Version, Run, or score.
## Fields, states, or lifecycle rules
- The Benchmark identity persists across changes to its current Dataset, coverage work, standards, and Harness candidates.
- Benchmark Datasets owns selected Cases and immutable Dataset Snapshots.
- A Benchmark Version fixes the evidence boundary used by a Run.
- Benchmark membership should be shaped by coverage work, not by whichever Cases are easiest to run.
- A Run result is weak if the Benchmark Version and saved Harness Version behind it are unclear.
- This page documents object semantics, not public execution, export, rate-limit, or API guarantees.
## Related objects
Benchmarks should be read with [Cases](/docs/assets/cases), [Policies](/docs/object-model/policies), [Rubrics](/docs/object-model/rubrics), [Coverage Engineering](/docs/coverage-engineering), and [Benchmark Evaluations](/docs/benchmark-evaluations). Use [Run an evaluation](/docs/benchmark-evaluations/run-evaluation) when the next step is execution.
{% example-demo title="Benchmarks boundary" %}
Raw case: A team refreshes coverage after finding unsupported compatibility claims.
Benchmark version: The refreshed version includes new unsupported-claim cases and the approved compatibility rubric.
Run: The candidate is evaluated against that version.
Interpretation: If the score drops, reviewers can see that the benchmark became harder instead of assuming the candidate behavior changed.
{% /example-demo %}
## Source confidence
Code-backed: the Benchmark type and workspace establish durable Benchmark identity; Benchmark Datasets → Snapshots establishes immutable Dataset boundaries; the evaluation-runs route consumes a specific Benchmark Version. The public object definition does not imply an execution or export API.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Benchmarks](/docs/coverage-engineering/benchmarks)
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Product quickstart](/docs/quickstart)
- [Task index](/docs/operating-manual/task-index)
{% /related-card-grid %}