Map External Evaluation Outputs

Prerequisites

• External outputs for the exact Benchmark Version.
• Durable Case IDs and, for repeated groups, an attempt-assignment plan.
• Permission to create or update the output-only Run.

Output mapping admits responses produced outside the managed Harness runtime as an output-only Run. Use upload or the displayed API path, then map every row to an immutable Case in the current Benchmark Version.

Row contract

The mapping template uses case_id, input, context, and output. Optional fields can carry latency, usage, and cost. case_id is the reliable join key; input and context help operators verify that the external row represents the intended immutable Case.

For repeated Run Groups, assign an attempt explicitly or use automatic assignment when the incoming rows can be distributed unambiguously. Never combine two external attempts into one output simply to satisfy the configured sample count.

Steps

1. Open Runs and start the external or reference-output mapping flow.
2. Download or inspect the template for the current Benchmark Version.
3. Populate exact Case IDs and outputs. Preserve the source system's telemetry only when it is measured.
4. Upload or submit through the displayed API workflow and review the preview.
5. Resolve unknown Cases, missing benchmark Cases, duplicates, or ambiguous attempt assignments.
6. Commit the mapping and inspect inserted, updated, missing, and unknown counts.
7. Follow evaluation progress and inspect the output-only Run from List.

Common mistakes

• Inventing Case IDs or joining only on input text.
• Reporting missing telemetry as zero.
• Mapping current editable Cases instead of the immutable Benchmark Version.
• Ignoring updated rows when the operation was expected to insert only.
• Assuming a successful upload proves that Rubric evaluation is complete.

Object and state changes

Committing inserts or updates mapped output rows and creates or updates the scoped output-only Run and attempt assignment. It does not create a Harness Version or modify immutable Cases.

Success criteria

• Every admitted row maps to the intended Case and attempt.
• Inserted, updated, missing, and unknown counts are understood.
• Evaluation completion remains separate from upload completion.

Common failure modes

• Joining on text while ignoring Case IDs.
• Overwriting an attempt unintentionally.
• Presenting the reference Run as an executable candidate.

Related reference pages

Related troubleshooting pages

Source confidence

Code-backed: the current output-mapping modal and Runs workspace define the row template, immutable Case mapping, attempt handling, result counts, and output-only Run boundary.