Skip to content

error

Benchmark run troubleshooting

Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely.

Benchmark run troubleshooting

Symptoms

  • The evaluation surface has no Benchmark Version to run.
  • Output import or mapping cannot identify a Case or output column.
  • A Run is created but remains empty, incomplete, or failed.
  • Results appear under the wrong candidate label or Harness Version.
  • Two Runs show a score change but do not share a comparable snapshot boundary.

Likely causes

  • No immutable Snapshot exists for the intended evaluation set.
  • Imported outputs are mapped to the wrong Case or column.
  • The saved Harness Version or candidate metadata does not match the evaluated system.
  • The Run is partial, failed, or being compared across different Benchmark Versions.

Check the boundary before the failure

  1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version.
  2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run.
  3. For managed execution, confirm the saved Harness Version and recorded settings. For imported results, confirm that the external candidate context describes the system that produced the submitted responses.
  4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior.
  5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates.

Fix

  • No runnable version: finish Case selection and create a Snapshot first.
  • Missing imported outputs: append only missing Cases to an uncompleted Trial using the exact manifest IDs. Correcting an accepted response or adding Cases after completion requires a new Trial in the external Run or a new Run. Preserve the original results.
  • Wrong Harness or metadata: create a correctly configured Run. Do not relabel completed results to represent a different system.
  • Partial managed failure: inspect and preserve the existing Run's recorded outcomes. For another independent execution under unchanged conditions, use Run again. Run evaluation reopens matching work, including failed work. Publish a new Harness Version when its code changes, or create a new Snapshot and Benchmark Version when Case content or criteria change.
  • Confusing comparison: compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed.

Prevention

Create a Snapshot before evaluation. For managed execution, save the exact Harness Version. For imported results, preserve external candidate context and preview the mapping on representative Cases. Before comparing, confirm that every difference between the two Runs is intentional and visible.

Recovery check

Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping.

Worked example

Example: scores drop after output import

A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue