Teammately Docs
Docs menu

reference

Reproducibility

Preserve enough source context to explain and repeat correctness decisions.

Reproducibility

Definition

Reproducibility means preserving enough exact identity and observable configuration to explain what was evaluated and to repeat the supported execution path. It does not mean every future execution will produce an identical stochastic output. It means a reader can distinguish changes in candidate, evidence, evaluator, sampling, and runtime metadata instead of attributing every result difference to the model.

Fields, states, or lifecycle rules

  • Preserve Project, Benchmark, Benchmark Version, Dataset Snapshot, and Run identity.
  • Preserve the exact saved Harness Version rather than an editable draft or display label.
  • Preserve admitted Case, Policy, and Rubric version boundaries through the Benchmark Version.
  • Record Run Group, attempt, sampling profile, evaluator set, and visible execution settings.
  • Retain run metadata and measured telemetry when captured; missing values remain unknown.
  • Record completeness, incomplete Cases, and terminal state beside scores.
  • Use canonical evaluation receipts for Improvement Session candidate claims.
  • Do not claim private worker activity, hidden reasoning, infrastructure internals, or unavailable traces as reproducibility evidence.

Dataset Snapshots preserve the evidence set. Benchmark Evaluations preserves candidate, Run, settings, results, and available telemetry. Compare and Arena interpret candidates inside compatible evidence boundaries. Improve adds Goal Contract, candidate, and canonical receipt identity when evaluation drives code or Harness changes.

Worked example

Diagnosing a score change

Two Runs use the same Harness Version but report different pass rates. The operator confirms that one Run used Benchmark Version 6 and the other used Version 7, which added source-conflict Cases and a revised grounding Rubric. The version and completeness record explains the movement. The team avoids filing a candidate regression until it compares Runs inside the same evidence boundary.

Source confidence

Code-backed: Snapshot, Run detail, and run-metadata surfaces expose the immutable evidence boundary, candidate identity, status, counts, settings, and available metadata needed for supported reproducibility. They do not promise deterministic model output or unrestricted execution traces.

Related task pages

Reference / Governance and ReproducibilityBenchmark VersioningPreserve benchmark snapshots so evidence can be compared across target and standard changes.Task / Benchmark EvaluationsCompare Harness VersionsCompare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.Concept / Benchmark EvaluationsBenchmark EvaluationsRun and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare.Quickstart / Learning ManualProduct quickstartConfigure one project foundation, one benchmark slice, one expert contribution, one evaluation, and one evidence-backed improvement.Concept / Operating ManualTask indexRoute correctness work to the current project foundation, benchmark workspace, evaluation, or improvement surface.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Governance and Reproducibility Benchmark Versioning Preserve benchmark snapshots so evidence can be compared across target and standard changes. Benchmark Evaluations Compare Harness Versions Compare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets. Object Model Benchmarks Understand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.
AI context