# Benchmark run troubleshooting Generated: 2026-09-13T04:39:33.136Z Source build: local Canonical docs: https://teammately.ai/docs --- id: troubleshooting.benchmark-runs title: Benchmark run troubleshooting summary: Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/benchmark-runs --- # Benchmark run troubleshooting ## Symptoms - The evaluation surface has no Benchmark Version to run. - Output import or mapping cannot identify a Case or output column. - A Run is created but remains empty, incomplete, or failed. - Results appear under the wrong candidate label or Harness Version. - Two Runs show a score change but do not share a comparable evidence boundary. ## Likely causes - No immutable Snapshot exists for the intended evidence set. - Imported outputs are mapped to the wrong Case or column. - The saved Harness Version or candidate metadata does not match the evaluated system. - The Run is partial, failed, or being compared across different Benchmark Versions. ## Check the boundary before the failure 1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version. 2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run. 3. In Run setup, confirm the saved Harness Version and candidate metadata describe the system that produced the outputs. 4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior. 5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates. ## Fix - **No runnable version:** finish Case selection and create a Snapshot first. - **No mapped outputs:** correct Case identifiers or column mapping, then submit again under the intended Run. - **Wrong Harness or metadata:** create a correctly configured Run. Do not relabel completed evidence to represent a different system. - **Partial failure:** preserve successful per-Case evidence when the product does, correct the failed input or execution boundary, and rerun using a clearly named attempt. - **Confusing comparison:** compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed. ## Prevention Create a Snapshot before execution, save the exact Harness Version, preview output mapping on representative Cases, and name candidate metadata consistently. Before comparing, confirm that every difference between the two Runs is intentional and visible. ## Recovery check Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping. {% example-demo title="Example: scores drop after output import" %} A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version. {% /example-demo %} ## Source confidence Code-backed: Runs index, setup, output mapping, Run detail, and metadata-display implementations establish the identifiers, mapping choices, and evidence shown during diagnosis. Backend-provider errors and customer Harness behavior may require additional operational logs outside this page. ## Related task pages {% related-card-grid title="Related workflows" %} - [Run an evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect results](/docs/benchmark-evaluations/inspect-results) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Run Metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Benchmarks and versions](/docs/concepts/benchmarks-and-versions) - [Benchmark Snapshots](/docs/coverage-engineering/benchmark-snapshots) - [Outputs](/docs/object-model/outputs) {% /related-card-grid %} --- id: benchmark-evaluations.run title: Run a Benchmark Evaluation summary: Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/run-evaluation --- # Run a Benchmark Evaluation Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready. ## Prerequisites - A Benchmark Version backed by the intended Dataset Snapshot. - Approved eligible Policies and Rubrics. - At least one saved project Harness Version. - Prepared Harness runtime and required secret grants. - A chosen number of Runs for each selected Harness. ## Steps 1. Open **Benchmark Evaluations** for the intended Benchmark Version. 2. Open Evaluation Settings if you need to adjust the machine configuration. 3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone. 4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume. 5. Supply any requested Run Metadata. Keep credentials out of descriptive fields. 6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one. 7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores. 8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable. ## Evidence created The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage. Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete. > No Draft execution > > A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching. {% example-demo title="Example: two candidates, three attempts" %} Harness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort. {% /example-demo %} ## Object and state changes Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort. ## Success criteria - Exact Harness and Benchmark Versions are recorded. - Each launch group contains the number of Runs requested for that Harness. - Output and evaluation progress reach an interpretable terminal state. - Incomplete or failed attempts remain visible. ## Common failure modes - Selecting the wrong saved Version or assuming Draft execution. - Reading partial evaluation as a complete cohort. - Treating absent telemetry as zero usage. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings) - [Harnesses](/docs/assets/harnesses) - [Run Metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Missing outputs](/docs/troubleshooting/missing-outputs) - [Authentication](/docs/troubleshooting/authentication) {% /related-card-grid %} ## Source confidence Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity. --- id: benchmark-evaluations.inspect title: Inspect Evaluation Results summary: Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/inspect-results --- # Inspect Evaluation Results ## Prerequisites - A visible Run or Run Group with output or evaluation progress. - Access to the exact Benchmark and Harness Version evidence. Start with completeness and identity, then move from aggregate signals to the Cases and evaluator failures that support them. ## Steps 1. Open Dashboard and confirm the Benchmark Version, candidate Harness Version, Run Group type, attempt count, and evaluation progress. 2. Read rankings with their metric family and uncertainty. Distinguish average score, passed at least once, and passed every time over observed Runs. Inspect Run counts and coverage; historical group-specific pass@n and pass^n retain their original meanings. 3. Open **List → Runs** to inspect group and Run status, output progress, evaluation progress, metadata, and available resource telemetry. 4. Open **List → Evaluation results** for the Case summary, outcome, failed Policies, failed Rubrics, and evaluated count. 5. Use Arena for pairwise disagreement or Compare for a Harness matrix across Cases, evaluator facts, or Coverage Facets. 6. Classify the next action as candidate work, evaluator clarification, Case correction, coverage work, external-output remapping, or no action. The List results surface is intentionally compact. Do not claim that it exposes full execution trajectories. The **Traces / Spans** segment currently reports a capability fence because the benchmark API does not provide evaluation execution traces. ## Reading incomplete and repeated evidence An aggregate calculated over fewer evaluable Cases can look better while covering less evidence. Record evaluated, incomplete, and missing counts before comparing candidates. For repeated groups, inspect whether the configured number of attempts exists for every candidate and whether one failed attempt changes the metric interpretation. Cost, tokens, and latency help route operational work but are nullable telemetry. Missing capture means unknown, not free or instantaneous execution. > Evaluator authority > > Policy and Rubric results are the correctness evidence admitted by the Benchmark Version. Rankings and telemetry summarize that evidence; they do not create a new standard. {% example-demo title="Example: apparent gain from incomplete evidence" %} Harness B leads the overall table, but List shows that twelve difficult Cases are still unevaluated for B. Arena also reports incomplete pairs. The operator waits for terminal evidence instead of starting Improve from a ranking that covers a smaller Case population. {% /example-demo %} ## Object and state changes Inspection, filtering, and navigation are read-only. Starting Improve, a Contribution, coverage work, or a later Run creates separate durable work while preserving the inspected evidence. ## Success criteria - Identity, completeness, metric family, and uncertainty are explicit. - Important signals resolve to Cases and admitted evaluator outcomes. - The next action targets the responsible artifact or candidate boundary. ## Common failure modes - Reporting rank without the evaluated population. - Inventing execution traces from the unavailable segment. - Starting candidate work when the Case or Rubric is wrong. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Dataset Snapshots](/docs/benchmark-datasets/snapshots) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Missing outputs](/docs/troubleshooting/missing-outputs) {% /related-card-grid %} ## Source confidence Code-backed: Dashboard, List, Run detail, and workspace types establish result summaries, completion, rankings, repeated metrics, telemetry, and the current trace capability fence. --- id: benchmark-evaluations.compare title: Compare Harness Versions summary: Compare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets. kind: task product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations/compare --- # Compare Harness Versions Compare shows aggregate observed evidence for saved Harness Versions. Choose the Versions, compatible evaluation configuration, and measurement to compare. Use **Inspect individual Run comparisons** for the detailed result matrix. ## Prerequisites Saved Harness Versions with evaluation results are shown for the selected Benchmark Version. Choose one or more Versions to inspect; Run counts may differ. An output-only imported reference Run cannot become a Harness column because it has no executable saved Version. ## Select a row mode The individual Run matrix offers row modes including Cases, all results, Policies, Rubrics, Dimension ontology values, Project Topics, Topic Groups, and Case Construction Patterns. Use Cases to inspect concrete disagreement, Policies or Rubrics to locate correctness movement, and Coverage Facets to see whether gains concentrate in one behavior slice. ## Steps 1. Confirm the immutable Benchmark Version and choose at least two visible Harness Versions. 2. Select an aggregate measurement, or open individual Run comparisons and choose a row mode. 3. Check evidence completeness for each Harness column. A blank or incomplete cell is not a failure. 4. Locate rows with material disagreement and connect them back to Case and evaluator evidence. 5. Preserve regressions and required-criterion failures next to gains. 6. Use the exact candidate and row evidence when starting an Improvement Session or requesting an Expert Contribution. Compare reads existing evidence and does not mutate Runs. Selecting aggregate Harness Versions recomputes their comparison on compatible evidence; individual matrix visibility is local presentation. It does not activate Harnesses or choose a winner. {% example-demo title="Example: facet-local improvement" %} Three Harness Versions look similar overall. The Project Topic row mode shows that Version 14 improves source-authority Topics but regresses escalation Topics. Switching to Cases identifies two regressions, and the team starts Improve with those exact failures instead of claiming a uniform improvement. {% /example-demo %} ## Common mistakes - Comparing different Benchmark Versions as though only the candidate moved. - Treating missing evidence as a failed cell. - Reading a facet aggregate without checking the distinct Cases behind it. - Describing an imported output-only Run as a Harness Version. - Selecting the newest Version solely because it is newest. ## Object and state changes Compare reads existing evidence. Selecting aggregate Versions reads their observed evidence; selecting individual matrix row modes changes presentation; it does not activate a Harness, mutate a Run, or retain a candidate. ## Success criteria - At least two exact Harness Versions share the same Benchmark Version. - Incomplete cells remain distinct from failed evidence. - Material movement resolves to Cases, evaluators, or Coverage Facets. ## Common failure modes - Comparing moved evidence boundaries as candidate-only change. - Treating local column visibility as product configuration. - Using an output-only Run as a Harness column. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Harnesses](/docs/assets/harnesses) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} ## Source confidence Code-backed: the active Compare route and Evaluation Matrix define Harness columns, local visibility, symmetric comparison, and the current evidence row modes.