Run a Benchmark Evaluation
Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready.
Prerequisites
- A Benchmark Version backed by the intended Dataset Snapshot.
- Approved eligible Policies and Rubrics.
- At least one saved project Harness Version.
- Prepared Harness runtime and required secret grants.
- A chosen number of Runs for each selected Harness.
Steps
- Open Benchmark Evaluations for the intended Benchmark Version.
- Open Evaluation Settings if you need to adjust the machine configuration.
- Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone.
- Choose the number of Runs for each Harness. Counts may differ; review the total execution volume.
- Supply any requested Run Metadata. Keep credentials out of descriptive fields.
- Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one.
- Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores.
- Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable.
Evidence created
The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage.
Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete.
Worked example
Example: two candidates, three attempts
Harness Versions 6 and 9 are active with n=3. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort.
Object and state changes
Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort.
Success criteria
- Exact Harness and Benchmark Versions are recorded.
- Each launch group contains the number of Runs requested for that Harness.
- Output and evaluation progress reach an interpretable terminal state.
- Incomplete or failed attempts remain visible.
Common failure modes
- Selecting the wrong saved Version or assuming Draft execution.
- Reading partial evaluation as a complete cohort.
- Treating absent telemetry as zero usage.
Related reference pages
Related troubleshooting pages
Source confidence
Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity.