Harnesses

Definition

A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements generate_result(request) at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.

A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.

Fields, states, or lifecycle rules

• A newly created Harness begins with one blank editable Draft.
• Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded.
• The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
• Draft validation errors block publication. Warnings remain visible for operator review.
• Publish creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also uses that Version for the currently selected benchmark.
• A previously saved Version can be selected with Use this Version for the current benchmark.
• Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
• Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
• Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
• Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
• Runtime delivery can include the rendered case view, static case_material_refs, and an optional supported world_instance_ref.
• Debug runs execute only a saved Version against a harness_contract.v1 request. They are authoring checks, not benchmark evaluation evidence.

Draft, saved version, and activation

Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.

The Evaluations execution settings show project Harnesses with saved Versions and control which are active for future managed Runs. Activation is benchmark-scoped. Every admitted Run still records the exact saved Harness Version, so later activation changes do not rewrite existing evidence.

Runtime, debug, and output mapping

Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.

Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness.

Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable.

Source confidence

Code-backed: the active Harness list and editor establish creation, archive state, Draft autosave and conflicts, validation, publication, activation, secrets, debug, Coding Agent proposals, and runtime preparation. The runnable adapter supports the execution boundary without making its internal API public.

Related task pages