# Map External Evaluation Outputs
Generated: 2026-09-13T04:39:00.334Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: benchmark-evaluations.output-mapping
title: Map External Evaluation Outputs
summary: Import reference outputs, map them to immutable benchmark Cases and attempts, and inspect the resulting output-only Run.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-08-22
canonical: /docs/benchmark-evaluations/output-mapping
---
# Map External Evaluation Outputs
## Prerequisites
- External outputs for the exact Benchmark Version.
- Durable Case IDs and, for repeated groups, an attempt-assignment plan.
- Permission to create or update the output-only Run.
Output mapping admits responses produced outside the managed Harness runtime as an output-only Run. Use upload or the displayed API path, then map every row to an immutable Case in the current Benchmark Version.
## Row contract
The mapping template uses `case_id`, `input`, `context`, and `output`. Optional fields can carry latency, usage, and cost. `case_id` is the reliable join key; input and context help operators verify that the external row represents the intended immutable Case.
For repeated Run Groups, assign an attempt explicitly or use automatic assignment when the incoming rows can be distributed unambiguously. Never combine two external attempts into one output simply to satisfy the configured sample count.
## Steps
1. Open Runs and start the external or reference-output mapping flow.
2. Download or inspect the template for the current Benchmark Version.
3. Populate exact Case IDs and outputs. Preserve the source system's telemetry only when it is measured.
4. Upload or submit through the displayed API workflow and review the preview.
5. Resolve unknown Cases, missing benchmark Cases, duplicates, or ambiguous attempt assignments.
6. Commit the mapping and inspect inserted, updated, missing, and unknown counts.
7. Follow evaluation progress and inspect the output-only Run from List.
> Reference output boundary
>
> An imported output-only Run can be scored and inspected, but it is not a saved Harness Version. It cannot be activated, optimized in Improve, or used as a Harness column in Compare or Arena.
## Common mistakes
- Inventing Case IDs or joining only on input text.
- Reporting missing telemetry as zero.
- Mapping current editable Cases instead of the immutable Benchmark Version.
- Ignoring updated rows when the operation was expected to insert only.
- Assuming a successful upload proves that Rubric evaluation is complete.
## Object and state changes
Committing inserts or updates mapped output rows and creates or updates the scoped output-only Run and attempt assignment. It does not create a Harness Version or modify immutable Cases.
## Success criteria
- Every admitted row maps to the intended Case and attempt.
- Inserted, updated, missing, and unknown counts are understood.
- Evaluation completion remains separate from upload completion.
## Common failure modes
- Joining on text while ignoring Case IDs.
- Overwriting an attempt unintentionally.
- Presenting the reference Run as an executable candidate.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
- [Connect model outputs](/docs/integrations/connect-model-outputs)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Output mapping](/docs/troubleshooting/output-mapping)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Dataset upload](/docs/troubleshooting/dataset-upload)
{% /related-card-grid %}
## Source confidence
Code-backed: the current output-mapping modal and Runs workspace define the row template, immutable Case mapping, attempt handling, result counts, and output-only Run boundary.
---
id: benchmark-evaluations.overview
title: Benchmark Evaluations
summary: Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare.
kind: concept
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations
---
# Benchmark Evaluations
Benchmark Evaluations is the version-scoped workspace for executing and comparing candidate systems. The active top-level tabs are **Dashboard**, **List**, **Arena**, and **Compare**. Every managed Run binds an exact saved Harness Version to the immutable Benchmark Version shown in the route.
> Evaluation boundary
>
> Interpret evidence inside its recorded Benchmark Version, Harness Version, Run or Run Group, evaluator set, and metadata. Run counts belong to launches. Additional launches add evidence without rewriting earlier Runs.
## Surfaces and objects
Dashboard summarizes progress, leaderboards, rank progression across Runs, and available resource telemetry. List is segmented into **Runs**, **Evaluation results**, and **Traces / Spans**. The results segment summarizes Case outcomes and Policy or Rubric failures. Arena compares candidate pairs across governed metrics. Compare is a symmetric matrix of Harness Versions across selected evidence rows.
A Run Group can collect one standard attempt or repeated attempts. A Run records one candidate execution and its per-Case progress. Evaluation results record the admitted Policy and Rubric outcomes. Costs, tokens, and latency are telemetry only when the provider or execution path captured them.
> Traces / Spans capability fence
>
> The List navigation exposes Traces / Spans, but the current benchmark API does not expose evaluation execution traces. Do not claim that trajectories, spans, private reasoning, or tool traces can be inspected from Benchmark Evaluations today.
## Decision checkpoint
| Need | Open | Evidence to preserve |
| --- | --- | --- |
| Configure and launch managed Runs | Evaluation Settings and New evaluation run | Machine, saved Harness Versions, and per-Harness Run counts |
| Start candidate execution | Run modal | Exact Harness and Benchmark Versions |
| Inspect status and output summaries | List → Runs or Evaluation results | Run Group, attempt, Case counts, incomplete state |
| Compare candidate pairs | Arena | Metric family, pair count, only-A, only-B, shared failures |
| Compare many candidates by governed rows | Compare | Harness columns and chosen Case or facet row mode |
| Admit external reference outputs | Output mapping | Case mapping, attempt assignment, insert/update report |
## Rankings and repeated sampling
Dashboard aggregates compatible observed Runs for each saved Harness Version across launches. Average score weights Runs equally. Supported binary views report passed at least once or passed every time over the observed case outcomes. Counts and missing evidence are shown; unequal counts do not prevent comparison. Historical group metrics retain their recorded meanings.
Ranking is a routing signal. A candidate can lead overall while failing required Policy or high-impact Rubric evidence. Use Arena or Compare to locate the disagreement and List to confirm completeness before starting Improve work.
## External outputs
Uploaded or API-supplied reference outputs create output-only Runs that can be scored and inspected in List. They are not saved Harness Versions and therefore cannot be optimized in Improve or selected as Harness columns in Compare or Arena.
{% example-demo title="Example: repeated evaluation without evidence drift" %}
A team launches three Runs of Harness Version 8 and one Run of Version 11 against the same Benchmark Version. Both appear with their evidence counts. A later launch of Version 11 adds two Runs to its aggregate evidence without changing either launch group. The team can inspect individual Runs before deciding whether more evidence is useful.
{% /example-demo %}
## Related workflows
{% related-card-grid title="Related workflows" %}
- [Configure evaluation execution](/docs/benchmark-evaluations/execution-settings)
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
- [Use Arena and rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Map external outputs](/docs/benchmark-evaluations/output-mapping)
{% /related-card-grid %}
## Source confidence
Code-backed: the active version-scoped workspace, settings, Run modal, List segments, Dashboard, Arena, and Compare routes define the current evaluation model and capability fences.
---
id: benchmark-evaluations.inspect
title: Inspect Evaluation Results
summary: Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/inspect-results
---
# Inspect Evaluation Results
## Prerequisites
- A visible Run or Run Group with output or evaluation progress.
- Access to the exact Benchmark and Harness Version evidence.
Start with completeness and identity, then move from aggregate signals to the Cases and evaluator failures that support them.
## Steps
1. Open Dashboard and confirm the Benchmark Version, candidate Harness Version, Run Group type, attempt count, and evaluation progress.
2. Read rankings with their metric family and uncertainty. Distinguish average score, passed at least once, and passed every time over observed Runs. Inspect Run counts and coverage; historical group-specific pass@n and pass^n retain their original meanings.
3. Open **List → Runs** to inspect group and Run status, output progress, evaluation progress, metadata, and available resource telemetry.
4. Open **List → Evaluation results** for the Case summary, outcome, failed Policies, failed Rubrics, and evaluated count.
5. Use Arena for pairwise disagreement or Compare for a Harness matrix across Cases, evaluator facts, or Coverage Facets.
6. Classify the next action as candidate work, evaluator clarification, Case correction, coverage work, external-output remapping, or no action.
The List results surface is intentionally compact. Do not claim that it exposes full execution trajectories. The **Traces / Spans** segment currently reports a capability fence because the benchmark API does not provide evaluation execution traces.
## Reading incomplete and repeated evidence
An aggregate calculated over fewer evaluable Cases can look better while covering less evidence. Record evaluated, incomplete, and missing counts before comparing candidates. For repeated groups, inspect whether the configured number of attempts exists for every candidate and whether one failed attempt changes the metric interpretation.
Cost, tokens, and latency help route operational work but are nullable telemetry. Missing capture means unknown, not free or instantaneous execution.
> Evaluator authority
>
> Policy and Rubric results are the correctness evidence admitted by the Benchmark Version. Rankings and telemetry summarize that evidence; they do not create a new standard.
{% example-demo title="Example: apparent gain from incomplete evidence" %}
Harness B leads the overall table, but List shows that twelve difficult Cases are still unevaluated for B. Arena also reports incomplete pairs. The operator waits for terminal evidence instead of starting Improve from a ranking that covers a smaller Case population.
{% /example-demo %}
## Object and state changes
Inspection, filtering, and navigation are read-only. Starting Improve, a Contribution, coverage work, or a later Run creates separate durable work while preserving the inspected evidence.
## Success criteria
- Identity, completeness, metric family, and uncertainty are explicit.
- Important signals resolve to Cases and admitted evaluator outcomes.
- The next action targets the responsible artifact or candidate boundary.
## Common failure modes
- Reporting rank without the evaluated population.
- Inventing execution traces from the unavailable segment.
- Starting candidate work when the Case or Rubric is wrong.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
{% /related-card-grid %}
## Source confidence
Code-backed: Dashboard, List, Run detail, and workspace types establish result summaries, completion, rankings, repeated metrics, telemetry, and the current trace capability fence.
---
id: benchmark-datasets.snapshots
title: Dataset Snapshots
summary: Freeze Cases, evaluator links, and representation facts as an immutable benchmark evidence boundary.
kind: task
product_area: benchmark_datasets
status: stable
updated: 2026-08-22
canonical: /docs/benchmark-datasets/snapshots
---
# Dataset Snapshots
## Prerequisites
- A reviewed current Case set.
- Approved eligible evaluator links and no Snapshot readiness blockers.
- Permission to create benchmark evidence.
A Dataset Snapshot freezes the benchmark's selected Cases, eligible evaluator links, and representation facts at a point in time. The live dataset remains editable; the Snapshot opens read-only **Cases** and **Representation** views.
## Create a Snapshot
The readiness check reports Case count, approved eligible Policy and Rubric counts, and blockers. Resolve every blocker before creation. Record a meaningful Snapshot label, then verify the displayed version, content hash, creation time, and Case count.
Creation does not make weak input trustworthy. Review Case clarity, coverage, materials, and evaluator applicability first. After creation, do not describe later mutable classifications or links as if they were part of the frozen state.
## Evidence rules
- Identify the exact Snapshot or resulting Benchmark Version in every Run and comparison.
- Create a new Snapshot when Case membership, material content, or admitted evaluator relationships change in a way that affects the claim.
- Do not mutate a Snapshot to “fix” historical evidence; correct the live dataset and freeze a new one.
- If historical Representation is unavailable, report that limitation instead of substituting current facts.
{% example-demo title="Example: preserving a coverage expansion" %}
After Case Review adds eight exception-handling Cases, the team verifies approved rubric links and creates a new Snapshot. Runs against the earlier Benchmark Version remain comparable within their old boundary, while new Runs explicitly use the expanded version.
{% /example-demo %}
## Object and state changes
Creation adds a new immutable Snapshot with its own label, version, hash, time, Case membership, evaluator links, and representation facts. It does not lock or copy edits back into the current dataset.
## Success criteria
- Readiness has no blockers.
- Identity fields and Case count match the intended boundary.
- Future Runs cite the resulting exact Benchmark Version.
## Common failure modes
- Snapshotting weak or invalid Cases because readiness passes structurally.
- Treating current classifications as part of an older Snapshot.
- Comparing candidates across moved Snapshot boundaries without disclosure.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Benchmark Datasets](/docs/benchmark-datasets)
- [Benchmark versioning](/docs/governance/benchmark-versioning)
- [Reproducibility](/docs/governance/reproducibility)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Dataset upload](/docs/troubleshooting/dataset-upload)
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
{% /related-card-grid %}
## Source confidence
Code-backed: the active Snapshots route defines readiness, blockers, immutable content, identity fields, and read-only Snapshot inspection.