# Output mapping troubleshooting

Generated: 2026-09-13T04:41:38.147Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="troubleshooting.output-mapping" slug="troubleshooting/output-mapping" url="https://teammately.ai/docs/troubleshooting/output-mapping.md" title="Output mapping troubleshooting" updated="2026-08-23" status="stable">
---
id: troubleshooting.output-mapping
title: Output mapping troubleshooting
summary: Fix external output rows that do not join cleanly to immutable Case IDs in an output-only Run.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-08-23
canonical: /docs/troubleshooting/output-mapping
---

# Output mapping troubleshooting

Use this when externally produced responses exist but the mapping preview cannot join them cleanly to the Cases in the selected Benchmark Version.

## Symptom

Rows are unmatched, duplicated, or mapped to the wrong Case ID, or the wrong column was selected as the response. No output-only Run should be trusted until the mapping preview is correct.

## Likely causes

- External rows use row order or display text instead of immutable Case IDs.
- The file contains Case IDs from another Benchmark Version or Dataset export.
- The selected response column or identifier column is wrong.
- Several candidate output sets were combined without distinct metadata.

## Diagnostic checks

- Compare several affected case IDs with the output source IDs.
- Check whether the output set has candidate metadata such as model, prompt, retrieval version, or run identity.
- Confirm the source file was produced from the same immutable Case population.
- Inspect duplicate, blank, and unknown identifier counts in the mapping preview.

## Fix

- Correct the external file and map using immutable Case IDs rather than row position.
- Select the intended response column and keep one candidate output set per output-only Run.
- Create a new output-only Run from the corrected mapping. Existing Runs remain evidence under their original mapping.
- Repair missing Case context through the Case workflow, not during output mapping; create a new version boundary when Case content changes.

## Prevention

- Preserve source IDs through import and output generation.
- Sample mapped outputs before expert contribution work and before Evaluation Runs.
- Keep candidate identity in run metadata.
- Treat each imported output-only Run as its own evidence record; it is not a saved Harness Version.

## Related task pages

{% related-card-grid title="Related task pages" %}
- [Importing cases](/docs/operating-manual/import-and-prepare-cases)
- [Map external outputs](/docs/benchmark-evaluations/output-mapping)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Target model outputs](/docs/object-model/outputs)
- [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
{% /related-card-grid %}

## Source confidence

Code-backed: the current output-mapping modal defines identifier and output-column selection, mapping preview, unmatched rows, metadata, and creation of an output-only Run. The Runs workspace keeps that imported evidence distinct from saved Harness Versions.
</doc>

<doc id="benchmark-evaluations.run" slug="benchmark-evaluations/run-evaluation" url="https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md" title="Run a Benchmark Evaluation" updated="2026-09-13" status="stable">
---
id: benchmark-evaluations.run
title: Run a Benchmark Evaluation
summary: Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/run-evaluation
---

# Run a Benchmark Evaluation

Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready.

## Prerequisites

- A Benchmark Version backed by the intended Dataset Snapshot.
- Approved eligible Policies and Rubrics.
- At least one saved project Harness Version.
- Prepared Harness runtime and required secret grants.
- A chosen number of Runs for each selected Harness.

## Steps

1. Open **Benchmark Evaluations** for the intended Benchmark Version.
2. Open Evaluation Settings if you need to adjust the machine configuration.
3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone.
4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume.
5. Supply any requested Run Metadata. Keep credentials out of descriptive fields.
6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one.
7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores.
8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable.

## Evidence created

The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage.

Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete.

> No Draft execution
>
> A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching.

{% example-demo title="Example: two candidates, three attempts" %}
Harness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort.
{% /example-demo %}

## Object and state changes

Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort.

## Success criteria

- Exact Harness and Benchmark Versions are recorded.
- Each launch group contains the number of Runs requested for that Harness.
- Output and evaluation progress reach an interpretable terminal state.
- Incomplete or failed attempts remain visible.

## Common failure modes

- Selecting the wrong saved Version or assuming Draft execution.
- Reading partial evaluation as a complete cohort.
- Treating absent telemetry as zero usage.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings)
- [Harnesses](/docs/assets/harnesses)
- [Run Metadata](/docs/benchmark-evaluations/run-metadata)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Authentication](/docs/troubleshooting/authentication)
{% /related-card-grid %}

## Source confidence

Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity.
</doc>

<doc id="benchmark-evaluations.output-mapping" slug="benchmark-evaluations/output-mapping" url="https://teammately.ai/docs/benchmark-evaluations/output-mapping.md" title="Map External Evaluation Outputs" updated="2026-08-22" status="stable">
---
id: benchmark-evaluations.output-mapping
title: Map External Evaluation Outputs
summary: Import reference outputs, map them to immutable benchmark Cases and attempts, and inspect the resulting output-only Run.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-08-22
canonical: /docs/benchmark-evaluations/output-mapping
---

# Map External Evaluation Outputs

## Prerequisites

- External outputs for the exact Benchmark Version.
- Durable Case IDs and, for repeated groups, an attempt-assignment plan.
- Permission to create or update the output-only Run.

Output mapping admits responses produced outside the managed Harness runtime as an output-only Run. Use upload or the displayed API path, then map every row to an immutable Case in the current Benchmark Version.

## Row contract

The mapping template uses `case_id`, `input`, `context`, and `output`. Optional fields can carry latency, usage, and cost. `case_id` is the reliable join key; input and context help operators verify that the external row represents the intended immutable Case.

For repeated Run Groups, assign an attempt explicitly or use automatic assignment when the incoming rows can be distributed unambiguously. Never combine two external attempts into one output simply to satisfy the configured sample count.

## Steps

1. Open Runs and start the external or reference-output mapping flow.
2. Download or inspect the template for the current Benchmark Version.
3. Populate exact Case IDs and outputs. Preserve the source system's telemetry only when it is measured.
4. Upload or submit through the displayed API workflow and review the preview.
5. Resolve unknown Cases, missing benchmark Cases, duplicates, or ambiguous attempt assignments.
6. Commit the mapping and inspect inserted, updated, missing, and unknown counts.
7. Follow evaluation progress and inspect the output-only Run from List.

> Reference output boundary
>
> An imported output-only Run can be scored and inspected, but it is not a saved Harness Version. It cannot be activated, optimized in Improve, or used as a Harness column in Compare or Arena.

## Common mistakes

- Inventing Case IDs or joining only on input text.
- Reporting missing telemetry as zero.
- Mapping current editable Cases instead of the immutable Benchmark Version.
- Ignoring updated rows when the operation was expected to insert only.
- Assuming a successful upload proves that Rubric evaluation is complete.

## Object and state changes

Committing inserts or updates mapped output rows and creates or updates the scoped output-only Run and attempt assignment. It does not create a Harness Version or modify immutable Cases.

## Success criteria

- Every admitted row maps to the intended Case and attempt.
- Inserted, updated, missing, and unknown counts are understood.
- Evaluation completion remains separate from upload completion.

## Common failure modes

- Joining on text while ignoring Case IDs.
- Overwriting an attempt unintentionally.
- Presenting the reference Run as an executable candidate.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
- [Connect model outputs](/docs/integrations/connect-model-outputs)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Output mapping](/docs/troubleshooting/output-mapping)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Dataset upload](/docs/troubleshooting/dataset-upload)
{% /related-card-grid %}

## Source confidence

Code-backed: the current output-mapping modal and Runs workspace define the row template, immutable Case mapping, attempt handling, result counts, and output-only Run boundary.
</doc>

<doc id="benchmark-evaluations.inspect" slug="benchmark-evaluations/inspect-results" url="https://teammately.ai/docs/benchmark-evaluations/inspect-results.md" title="Inspect Evaluation Results" updated="2026-09-13" status="stable">
---
id: benchmark-evaluations.inspect
title: Inspect Evaluation Results
summary: Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/inspect-results
---

# Inspect Evaluation Results

## Prerequisites

- A visible Run or Run Group with output or evaluation progress.
- Access to the exact Benchmark and Harness Version evidence.

Start with completeness and identity, then move from aggregate signals to the Cases and evaluator failures that support them.

## Steps

1. Open Dashboard and confirm the Benchmark Version, candidate Harness Version, Run Group type, attempt count, and evaluation progress.
2. Read rankings with their metric family and uncertainty. Distinguish average score, passed at least once, and passed every time over observed Runs. Inspect Run counts and coverage; historical group-specific pass@n and pass^n retain their original meanings.
3. Open **List → Runs** to inspect group and Run status, output progress, evaluation progress, metadata, and available resource telemetry.
4. Open **List → Evaluation results** for the Case summary, outcome, failed Policies, failed Rubrics, and evaluated count.
5. Use Arena for pairwise disagreement or Compare for a Harness matrix across Cases, evaluator facts, or Coverage Facets.
6. Classify the next action as candidate work, evaluator clarification, Case correction, coverage work, external-output remapping, or no action.

The List results surface is intentionally compact. Do not claim that it exposes full execution trajectories. The **Traces / Spans** segment currently reports a capability fence because the benchmark API does not provide evaluation execution traces.

## Reading incomplete and repeated evidence

An aggregate calculated over fewer evaluable Cases can look better while covering less evidence. Record evaluated, incomplete, and missing counts before comparing candidates. For repeated groups, inspect whether the configured number of attempts exists for every candidate and whether one failed attempt changes the metric interpretation.

Cost, tokens, and latency help route operational work but are nullable telemetry. Missing capture means unknown, not free or instantaneous execution.

> Evaluator authority
>
> Policy and Rubric results are the correctness evidence admitted by the Benchmark Version. Rankings and telemetry summarize that evidence; they do not create a new standard.

{% example-demo title="Example: apparent gain from incomplete evidence" %}
Harness B leads the overall table, but List shows that twelve difficult Cases are still unevaluated for B. Arena also reports incomplete pairs. The operator waits for terminal evidence instead of starting Improve from a ranking that covers a smaller Case population.
{% /example-demo %}

## Object and state changes

Inspection, filtering, and navigation are read-only. Starting Improve, a Contribution, coverage work, or a later Run creates separate durable work while preserving the inspected evidence.

## Success criteria

- Identity, completeness, metric family, and uncertainty are explicit.
- Important signals resolve to Cases and admitted evaluator outcomes.
- The next action targets the responsible artifact or candidate boundary.

## Common failure modes

- Reporting rank without the evaluated population.
- Inventing execution traces from the unavailable segment.
- Starting candidate work when the Case or Rubric is wrong.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
{% /related-card-grid %}

## Source confidence

Code-backed: Dashboard, List, Run detail, and workspace types establish result summaries, completion, rankings, repeated metrics, telemetry, and the current trace capability fence.
</doc>