# Connect Model Outputs

Generated: 2026-09-13T04:41:39.634Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="integrations.connect-model-outputs" slug="integrations/connect-model-outputs" url="https://teammately.ai/docs/integrations/connect-model-outputs.md" title="Connect Model Outputs" updated="2026-08-22" status="stable">
---
id: integrations.connect-model-outputs
title: Connect Model Outputs
summary: Map externally produced outputs to immutable Benchmark Cases and create an output-only reference Run.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-08-22
canonical: /docs/integrations/connect-model-outputs
---

# Connect Model Outputs

## Prerequisites

- An immutable Benchmark Version containing the Cases to evaluate.
- One externally produced output per required Case.
- Durable Case IDs from that Benchmark Version.
- A reference label that identifies the external system or candidate.
- Measured latency, usage, or cost only when the source system actually recorded it.

## Before and after

| before | after |
| --- | --- |
| Outputs exist in an external file or system | Outputs are mapped to exact immutable Benchmark Cases |
| Candidate identity is customer-owned context | A Teammately reference Run preserves both its own ID and the external reference label |
| No Teammately evaluation state exists | Rubric evaluation proceeds as a separate Run lifecycle |
| Missing telemetry may be ambiguous | Unmeasured telemetry remains absent rather than becoming zero |

## Map outputs

### Task steps: Connect external model outputs

1. Open the intended Benchmark Version and go to **Benchmark Evaluations** → **Runs**.
2. Choose **Import reference outputs** and name the external system or candidate clearly.
3. Download the mapping template for the current Benchmark Version. Keep `case_id` unchanged; use input and context columns only to verify the match.
4. Populate `output` for each Case. Add latency, usage, or cost columns only for values measured by the producing system.
5. Upload the file and inspect unknown Case IDs, missing Benchmark Cases, duplicates, and output previews.
6. Resolve every mapping error. Do not join on input text or force an output onto a similar-looking Case.
7. Confirm the mapping and inspect the output-only reference Run.
8. Wait for evaluation to complete before interpreting result summaries or failures.

## Object and state changes

The workflow creates an ordinary output-only reference Run for one immutable Benchmark Version and associates submitted outputs with its Cases. It can also store measured telemetry supplied with those outputs.

It does not create or save a Harness Version, change Benchmark membership, mutate Cases, approve reference responses, or make the external candidate available to Improve as an executable Harness.

## Success criteria

- Every output joins through the exact Case ID from the intended Benchmark Version.
- The reference label distinguishes this output set from other Runs.
- Unknown, missing, duplicate, inserted, and updated counts are understood before interpretation.
- Unmeasured telemetry is absent.
- The output-only Run is not presented as a managed Harness candidate.
- Result interpretation waits for evaluation completion.

## Common failure modes

- Reusing IDs from the editable Project Case collection instead of the frozen Benchmark Version.
- Joining on input text, row order, or a customer ID without verifying the Teammately Case ID.
- Uploading outputs for two candidate versions under one reference label.
- Reporting missing latency or cost as zero.
- Treating successful mapping as successful evaluation.
- Assuming the reference Run can enter Harness Compare, Arena, or Improve as an executable candidate.

{% example-demo title="Retrieval candidate outputs" %}
A retrieval team evaluates a new indexing configuration outside Teammately. It exports one answer per frozen Benchmark Case and preserves its own generation ID. In the mapping template, each answer joins on `case_id`; the generation ID remains correlation context and measured latency is included.

The imported set becomes a reference Run. Teammately evaluates its outputs against the Benchmark Version's Policy and Rubric evidence, while the indexing configuration itself remains outside Teammately as a non-Harness system.
{% /example-demo %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Map External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
- [Integrations](/docs/integrations)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Output mapping](/docs/troubleshooting/output-mapping)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Benchmark results changed](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
{% /related-card-grid %}

## Source confidence

Code-backed: the current external-reference Run and output-mapping surfaces define the immutable Case join, template fields, reference-Run boundary, and result transition. Exact customer API serialization remains publication-gated separately.
</doc>

<doc id="benchmark-evaluations.output-mapping" slug="benchmark-evaluations/output-mapping" url="https://teammately.ai/docs/benchmark-evaluations/output-mapping.md" title="Map External Evaluation Outputs" updated="2026-08-22" status="stable">
---
id: benchmark-evaluations.output-mapping
title: Map External Evaluation Outputs
summary: Import reference outputs, map them to immutable benchmark Cases and attempts, and inspect the resulting output-only Run.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-08-22
canonical: /docs/benchmark-evaluations/output-mapping
---

# Map External Evaluation Outputs

## Prerequisites

- External outputs for the exact Benchmark Version.
- Durable Case IDs and, for repeated groups, an attempt-assignment plan.
- Permission to create or update the output-only Run.

Output mapping admits responses produced outside the managed Harness runtime as an output-only Run. Use upload or the displayed API path, then map every row to an immutable Case in the current Benchmark Version.

## Row contract

The mapping template uses `case_id`, `input`, `context`, and `output`. Optional fields can carry latency, usage, and cost. `case_id` is the reliable join key; input and context help operators verify that the external row represents the intended immutable Case.

For repeated Run Groups, assign an attempt explicitly or use automatic assignment when the incoming rows can be distributed unambiguously. Never combine two external attempts into one output simply to satisfy the configured sample count.

## Steps

1. Open Runs and start the external or reference-output mapping flow.
2. Download or inspect the template for the current Benchmark Version.
3. Populate exact Case IDs and outputs. Preserve the source system's telemetry only when it is measured.
4. Upload or submit through the displayed API workflow and review the preview.
5. Resolve unknown Cases, missing benchmark Cases, duplicates, or ambiguous attempt assignments.
6. Commit the mapping and inspect inserted, updated, missing, and unknown counts.
7. Follow evaluation progress and inspect the output-only Run from List.

> Reference output boundary
>
> An imported output-only Run can be scored and inspected, but it is not a saved Harness Version. It cannot be activated, optimized in Improve, or used as a Harness column in Compare or Arena.

## Common mistakes

- Inventing Case IDs or joining only on input text.
- Reporting missing telemetry as zero.
- Mapping current editable Cases instead of the immutable Benchmark Version.
- Ignoring updated rows when the operation was expected to insert only.
- Assuming a successful upload proves that Rubric evaluation is complete.

## Object and state changes

Committing inserts or updates mapped output rows and creates or updates the scoped output-only Run and attempt assignment. It does not create a Harness Version or modify immutable Cases.

## Success criteria

- Every admitted row maps to the intended Case and attempt.
- Inserted, updated, missing, and unknown counts are understood.
- Evaluation completion remains separate from upload completion.

## Common failure modes

- Joining on text while ignoring Case IDs.
- Overwriting an attempt unintentionally.
- Presenting the reference Run as an executable candidate.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
- [Connect model outputs](/docs/integrations/connect-model-outputs)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Output mapping](/docs/troubleshooting/output-mapping)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Dataset upload](/docs/troubleshooting/dataset-upload)
{% /related-card-grid %}

## Source confidence

Code-backed: the current output-mapping modal and Runs workspace define the row template, immutable Case mapping, attempt handling, result counts, and output-only Run boundary.
</doc>

<doc id="benchmark-evaluations.run" slug="benchmark-evaluations/run-evaluation" url="https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md" title="Run a Benchmark Evaluation" updated="2026-09-13" status="stable">
---
id: benchmark-evaluations.run
title: Run a Benchmark Evaluation
summary: Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/run-evaluation
---

# Run a Benchmark Evaluation

Launch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready.

## Prerequisites

- A Benchmark Version backed by the intended Dataset Snapshot.
- Approved eligible Policies and Rubrics.
- At least one saved project Harness Version.
- Prepared Harness runtime and required secret grants.
- A chosen number of Runs for each selected Harness.

## Steps

1. Open **Benchmark Evaluations** for the intended Benchmark Version.
2. Open Evaluation Settings if you need to adjust the machine configuration.
3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone.
4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume.
5. Supply any requested Run Metadata. Keep credentials out of descriptive fields.
6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one.
7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores.
8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable.

## Evidence created

The launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage.

Dashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete.

> No Draft execution
>
> A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching.

{% example-demo title="Example: two candidates, three attempts" %}
Harness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort.
{% /example-demo %}

## Object and state changes

Launching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort.

## Success criteria

- Exact Harness and Benchmark Versions are recorded.
- Each launch group contains the number of Runs requested for that Harness.
- Output and evaluation progress reach an interpretable terminal state.
- Incomplete or failed attempts remain visible.

## Common failure modes

- Selecting the wrong saved Version or assuming Draft execution.
- Reading partial evaluation as a complete cohort.
- Treating absent telemetry as zero usage.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings)
- [Harnesses](/docs/assets/harnesses)
- [Run Metadata](/docs/benchmark-evaluations/run-metadata)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
- [Authentication](/docs/troubleshooting/authentication)
{% /related-card-grid %}

## Source confidence

Code-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity.
</doc>

<doc id="benchmark-datasets.snapshots" slug="benchmark-datasets/snapshots" url="https://teammately.ai/docs/benchmark-datasets/snapshots.md" title="Dataset Snapshots" updated="2026-08-22" status="stable">
---
id: benchmark-datasets.snapshots
title: Dataset Snapshots
summary: Freeze Cases, evaluator links, and representation facts as an immutable benchmark evidence boundary.
kind: task
product_area: benchmark_datasets
status: stable
updated: 2026-08-22
canonical: /docs/benchmark-datasets/snapshots
---

# Dataset Snapshots

## Prerequisites

- A reviewed current Case set.
- Approved eligible evaluator links and no Snapshot readiness blockers.
- Permission to create benchmark evidence.

A Dataset Snapshot freezes the benchmark's selected Cases, eligible evaluator links, and representation facts at a point in time. The live dataset remains editable; the Snapshot opens read-only **Cases** and **Representation** views.

## Create a Snapshot

The readiness check reports Case count, approved eligible Policy and Rubric counts, and blockers. Resolve every blocker before creation. Record a meaningful Snapshot label, then verify the displayed version, content hash, creation time, and Case count.

Creation does not make weak input trustworthy. Review Case clarity, coverage, materials, and evaluator applicability first. After creation, do not describe later mutable classifications or links as if they were part of the frozen state.

## Evidence rules

- Identify the exact Snapshot or resulting Benchmark Version in every Run and comparison.
- Create a new Snapshot when Case membership, material content, or admitted evaluator relationships change in a way that affects the claim.
- Do not mutate a Snapshot to “fix” historical evidence; correct the live dataset and freeze a new one.
- If historical Representation is unavailable, report that limitation instead of substituting current facts.

{% example-demo title="Example: preserving a coverage expansion" %}
After Case Review adds eight exception-handling Cases, the team verifies approved rubric links and creates a new Snapshot. Runs against the earlier Benchmark Version remain comparable within their old boundary, while new Runs explicitly use the expanded version.
{% /example-demo %}

## Object and state changes

Creation adds a new immutable Snapshot with its own label, version, hash, time, Case membership, evaluator links, and representation facts. It does not lock or copy edits back into the current dataset.

## Success criteria

- Readiness has no blockers.
- Identity fields and Case count match the intended boundary.
- Future Runs cite the resulting exact Benchmark Version.

## Common failure modes

- Snapshotting weak or invalid Cases because readiness passes structurally.
- Treating current classifications as part of an older Snapshot.
- Comparing candidates across moved Snapshot boundaries without disclosure.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmark Datasets](/docs/benchmark-datasets)
- [Benchmark versioning](/docs/governance/benchmark-versioning)
- [Reproducibility](/docs/governance/reproducibility)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Dataset upload](/docs/troubleshooting/dataset-upload)
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
{% /related-card-grid %}

## Source confidence

Code-backed: the active Snapshots route defines readiness, blockers, immutable content, identity fields, and read-only Snapshot inspection.
</doc>

<doc id="troubleshooting.output-mapping" slug="troubleshooting/output-mapping" url="https://teammately.ai/docs/troubleshooting/output-mapping.md" title="Output mapping troubleshooting" updated="2026-08-23" status="stable">
---
id: troubleshooting.output-mapping
title: Output mapping troubleshooting
summary: Fix external output rows that do not join cleanly to immutable Case IDs in an output-only Run.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-08-23
canonical: /docs/troubleshooting/output-mapping
---

# Output mapping troubleshooting

Use this when externally produced responses exist but the mapping preview cannot join them cleanly to the Cases in the selected Benchmark Version.

## Symptom

Rows are unmatched, duplicated, or mapped to the wrong Case ID, or the wrong column was selected as the response. No output-only Run should be trusted until the mapping preview is correct.

## Likely causes

- External rows use row order or display text instead of immutable Case IDs.
- The file contains Case IDs from another Benchmark Version or Dataset export.
- The selected response column or identifier column is wrong.
- Several candidate output sets were combined without distinct metadata.

## Diagnostic checks

- Compare several affected case IDs with the output source IDs.
- Check whether the output set has candidate metadata such as model, prompt, retrieval version, or run identity.
- Confirm the source file was produced from the same immutable Case population.
- Inspect duplicate, blank, and unknown identifier counts in the mapping preview.

## Fix

- Correct the external file and map using immutable Case IDs rather than row position.
- Select the intended response column and keep one candidate output set per output-only Run.
- Create a new output-only Run from the corrected mapping. Existing Runs remain evidence under their original mapping.
- Repair missing Case context through the Case workflow, not during output mapping; create a new version boundary when Case content changes.

## Prevention

- Preserve source IDs through import and output generation.
- Sample mapped outputs before expert contribution work and before Evaluation Runs.
- Keep candidate identity in run metadata.
- Treat each imported output-only Run as its own evidence record; it is not a saved Harness Version.

## Related task pages

{% related-card-grid title="Related task pages" %}
- [Importing cases](/docs/operating-manual/import-and-prepare-cases)
- [Map external outputs](/docs/benchmark-evaluations/output-mapping)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Target model outputs](/docs/object-model/outputs)
- [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
{% /related-card-grid %}

## Source confidence

Code-backed: the current output-mapping modal defines identifier and output-column selection, mapping preview, unmatched rows, metadata, and creation of an output-only Run. The Runs workspace keeps that imported evidence distinct from saved Harness Versions.
</doc>