---
id: benchmark-evaluations.overview
title: Benchmark Evaluations
summary: Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare.
kind: concept
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations
---

# Benchmark Evaluations

Benchmark Evaluations is the version-scoped workspace for executing and comparing candidate systems. The active top-level tabs are **Dashboard**, **List**, **Arena**, and **Compare**. Every managed Run binds an exact saved Harness Version to the immutable Benchmark Version shown in the route.

> Evaluation boundary
>
> Interpret evidence inside its recorded Benchmark Version, Harness Version, Run or Run Group, evaluator set, and metadata. Run counts belong to launches. Additional launches add evidence without rewriting earlier Runs.

## Surfaces and objects

Dashboard summarizes progress, leaderboards, rank progression across Runs, and available resource telemetry. List is segmented into **Runs**, **Evaluation results**, and **Traces / Spans**. The results segment summarizes Case outcomes and Policy or Rubric failures. Arena compares candidate pairs across governed metrics. Compare is a symmetric matrix of Harness Versions across selected evidence rows.

A Run Group can collect one standard attempt or repeated attempts. A Run records one candidate execution and its per-Case progress. Evaluation results record the admitted Policy and Rubric outcomes. Costs, tokens, and latency are telemetry only when the provider or execution path captured them.

> Traces / Spans capability fence
>
> The List navigation exposes Traces / Spans, but the current benchmark API does not expose evaluation execution traces. Do not claim that trajectories, spans, private reasoning, or tool traces can be inspected from Benchmark Evaluations today.

## Decision checkpoint

| Need | Open | Evidence to preserve |
| --- | --- | --- |
| Configure and launch managed Runs | Evaluation Settings and New evaluation run | Machine, saved Harness Versions, and per-Harness Run counts |
| Start candidate execution | Run modal | Exact Harness and Benchmark Versions |
| Inspect status and output summaries | List → Runs or Evaluation results | Run Group, attempt, Case counts, incomplete state |
| Compare candidate pairs | Arena | Metric family, pair count, only-A, only-B, shared failures |
| Compare many candidates by governed rows | Compare | Harness columns and chosen Case or facet row mode |
| Admit external reference outputs | Output mapping | Case mapping, attempt assignment, insert/update report |

## Rankings and repeated sampling

Dashboard aggregates compatible observed Runs for each saved Harness Version across launches. Average score weights Runs equally. Supported binary views report passed at least once or passed every time over the observed case outcomes. Counts and missing evidence are shown; unequal counts do not prevent comparison. Historical group metrics retain their recorded meanings.

Ranking is a routing signal. A candidate can lead overall while failing required Policy or high-impact Rubric evidence. Use Arena or Compare to locate the disagreement and List to confirm completeness before starting Improve work.

## External outputs

Uploaded or API-supplied reference outputs create output-only Runs that can be scored and inspected in List. They are not saved Harness Versions and therefore cannot be optimized in Improve or selected as Harness columns in Compare or Arena.

{% example-demo title="Example: repeated evaluation without evidence drift" %}
A team launches three Runs of Harness Version 8 and one Run of Version 11 against the same Benchmark Version. Both appear with their evidence counts. A later launch of Version 11 adds two Runs to its aggregate evidence without changing either launch group. The team can inspect individual Runs before deciding whether more evidence is useful.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Configure evaluation execution](/docs/benchmark-evaluations/execution-settings)
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
- [Use Arena and rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Map external outputs](/docs/benchmark-evaluations/output-mapping)
{% /related-card-grid %}

## Source confidence

Code-backed: the active version-scoped workspace, settings, Run modal, List segments, Dashboard, Arena, and Compare routes define the current evaluation model and capability fences.