# Evaluation Execution Settings
Generated: 2026-09-13T04:33:15.146Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: benchmark-evaluations.execution-settings
title: Evaluation Execution Settings
summary: Configure machines and choose saved Harness Versions and Run counts at launch.
kind: reference
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/execution-settings
---
# Evaluation Execution Settings
## Definition
Evaluation Settings contains machine settings for managed execution. Harness selection and repetition are choices made when launching an evaluation.
## Fields, states, or lifecycle rules
- Choose an exact saved project Harness Version when launching. Benchmark activation is not required.
- Choose a whole number of Runs from 1 through 50 for each selected Harness. The default is one, and different Harnesses may have different counts.
- Each managed Harness launch creates one Run Group containing the requested Runs, including a one-Run group.
- Existing Runs keep their immutable configuration and evidence. Launching more Runs creates another group rather than rewriting previous membership.
## Reliability choices
Dashboard, Compare, and Arena aggregate compatible observed Runs for each saved Harness Version across launch groups. Average score weights individual Runs equally. Passed at least once and passed every time summarize observed binary case outcomes where the evaluation framework supports them.
Run counts can differ. The product displays the counts and coverage and warns about unequal evidence without requiring another launch. Missing or infrastructure-failed observations are not numerical successes or failures.
## Before launching
Check the exact saved Version, machine selection, and requested execution volume. Runtime preparation and required access still apply. Imported external outputs remain a separate flow because they have no executable Harness Version.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Manage Harnesses](/docs/assets/harnesses)
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
{% /related-card-grid %}
## Source confidence
Code-backed: Evaluation Settings defines machine configuration; the shared Run modal defines saved-Version selection and per-launch Run counts.
---
id: benchmark-evaluations.overview
title: Benchmark Evaluations
summary: Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare.
kind: concept
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations
---
# Benchmark Evaluations
Benchmark Evaluations is the version-scoped workspace for executing and comparing candidate systems. The active top-level tabs are **Dashboard**, **List**, **Arena**, and **Compare**. Every managed Run binds an exact saved Harness Version to the immutable Benchmark Version shown in the route.
> Evaluation boundary
>
> Interpret evidence inside its recorded Benchmark Version, Harness Version, Run or Run Group, evaluator set, and metadata. Run counts belong to launches. Additional launches add evidence without rewriting earlier Runs.
## Surfaces and objects
Dashboard summarizes progress, leaderboards, rank progression across Runs, and available resource telemetry. List is segmented into **Runs**, **Evaluation results**, and **Traces / Spans**. The results segment summarizes Case outcomes and Policy or Rubric failures. Arena compares candidate pairs across governed metrics. Compare is a symmetric matrix of Harness Versions across selected evidence rows.
A Run Group can collect one standard attempt or repeated attempts. A Run records one candidate execution and its per-Case progress. Evaluation results record the admitted Policy and Rubric outcomes. Costs, tokens, and latency are telemetry only when the provider or execution path captured them.
> Traces / Spans capability fence
>
> The List navigation exposes Traces / Spans, but the current benchmark API does not expose evaluation execution traces. Do not claim that trajectories, spans, private reasoning, or tool traces can be inspected from Benchmark Evaluations today.
## Decision checkpoint
| Need | Open | Evidence to preserve |
| --- | --- | --- |
| Configure and launch managed Runs | Evaluation Settings and New evaluation run | Machine, saved Harness Versions, and per-Harness Run counts |
| Start candidate execution | Run modal | Exact Harness and Benchmark Versions |
| Inspect status and output summaries | List → Runs or Evaluation results | Run Group, attempt, Case counts, incomplete state |
| Compare candidate pairs | Arena | Metric family, pair count, only-A, only-B, shared failures |
| Compare many candidates by governed rows | Compare | Harness columns and chosen Case or facet row mode |
| Admit external reference outputs | Output mapping | Case mapping, attempt assignment, insert/update report |
## Rankings and repeated sampling
Dashboard aggregates compatible observed Runs for each saved Harness Version across launches. Average score weights Runs equally. Supported binary views report passed at least once or passed every time over the observed case outcomes. Counts and missing evidence are shown; unequal counts do not prevent comparison. Historical group metrics retain their recorded meanings.
Ranking is a routing signal. A candidate can lead overall while failing required Policy or high-impact Rubric evidence. Use Arena or Compare to locate the disagreement and List to confirm completeness before starting Improve work.
## External outputs
Uploaded or API-supplied reference outputs create output-only Runs that can be scored and inspected in List. They are not saved Harness Versions and therefore cannot be optimized in Improve or selected as Harness columns in Compare or Arena.
{% example-demo title="Example: repeated evaluation without evidence drift" %}
A team launches three Runs of Harness Version 8 and one Run of Version 11 against the same Benchmark Version. Both appear with their evidence counts. A later launch of Version 11 adds two Runs to its aggregate evidence without changing either launch group. The team can inspect individual Runs before deciding whether more evidence is useful.
{% /example-demo %}
## Related workflows
{% related-card-grid title="Related workflows" %}
- [Configure evaluation execution](/docs/benchmark-evaluations/execution-settings)
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
- [Use Arena and rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Map external outputs](/docs/benchmark-evaluations/output-mapping)
{% /related-card-grid %}
## Source confidence
Code-backed: the active version-scoped workspace, settings, Run modal, List segments, Dashboard, Arena, and Compare routes define the current evaluation model and capability fences.
---
id: benchmark-evaluations.run
title: Run a Benchmark Evaluation
summary: Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/run-evaluation
---
# Run a Benchmark Evaluation
Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready.
## Prerequisites
- A Benchmark Version backed by the intended Dataset Snapshot.
- Approved eligible Policies and Rubrics.
- At least one saved project Harness Version.
- Prepared Harness runtime and required secret grants.
- A chosen number of Runs for each selected Harness.
## Steps
1. Open **Benchmark Evaluations** for the intended Benchmark Version.
2. Open Evaluation Settings if you need to adjust the machine configuration.
3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone.
4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume.
5. Supply any requested Run Metadata. Keep credentials out of descriptive fields.
6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one.
7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores.
8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable.
## Evidence created
The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage.
Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete.
> No Draft execution
>
> A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching.
{% example-demo title="Example: two candidates, three attempts" %}
Harness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort.
{% /example-demo %}
## Object and state changes
Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort.
## Success criteria
- Exact Harness and Benchmark Versions are recorded.
- Each launch group contains the number of Runs requested for that Harness.
- Output and evaluation progress reach an interpretable terminal state.
- Incomplete or failed attempts remain visible.
## Common failure modes
- Selecting the wrong saved Version or assuming Draft execution.
- Reading partial evaluation as a complete cohort.
- Treating absent telemetry as zero usage.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings)
- [Harnesses](/docs/assets/harnesses)
- [Run Metadata](/docs/benchmark-evaluations/run-metadata)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Authentication](/docs/troubleshooting/authentication)
{% /related-card-grid %}
## Source confidence
Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity.
---
id: assets.harnesses
title: Harnesses
summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
kind: reference
product_area: assets
status: stable
updated: 2026-08-22
canonical: /docs/assets/harnesses
---
# Harnesses
## Definition
A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.
A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.
## Fields, states, or lifecycle rules
- A newly created Harness begins with one blank editable Draft.
- Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded.
- The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
- Draft validation errors block publication. Warnings remain visible for operator review.
- **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also uses that Version for the currently selected benchmark.
- A previously saved Version can be selected with **Use this Version** for the current benchmark.
- Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
- Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
- Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
- Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
- Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`.
- Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation evidence.
## Draft, saved version, and activation
Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.
The Evaluations execution settings show project Harnesses with saved Versions and control which are active for future managed Runs. Activation is benchmark-scoped. Every admitted Run still records the exact saved Harness Version, so later activation changes do not rewrite existing evidence.
## Runtime, debug, and output mapping
Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.
Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness.
Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable.
{% example-demo title="Example: exact candidate identity" %}
An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work.
{% /example-demo %}
## Source confidence
Code-backed: the active Harness list and editor establish creation, archive state, Draft autosave and conflicts, validation, publication, activation, secrets, debug, Coding Agent proposals, and runtime preparation. The runnable adapter supports the execution boundary without making its internal API public.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Start an Improvement Session](/docs/improve/start-improvement-session)
- [Configure Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}