# Missing Outputs Generated: 2026-09-13T04:38:59.937Z Source build: local Canonical docs: https://teammately.ai/docs --- id: troubleshooting.missing-outputs title: Missing Outputs summary: Separate a missing managed Run response from an unmatched row in an imported output-only Run. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/missing-outputs --- # Missing Outputs Use this when an Evaluation Run has no response for an intended Case. First identify whether the evidence came from managed Harness execution or an imported output-only Run. ## Symptom A managed Run is incomplete or has a failed or unevaluable Case attempt, or an imported output-only Run shows an unmatched Case row. An input-only Case by itself is not an error: managed execution is expected to generate its response. ## Likely causes - Managed execution failed, stopped, or could not evaluate the Case. - The wrong saved Harness Version or execution settings were selected. - An external output row has a missing, unknown, or duplicate immutable Case ID. - The imported file omitted Cases that the output-only Run was intended to cover. ## Diagnostic checks - Open the Run and check its lifecycle, attempts, failed Cases, and selected Harness Version. - Confirm that the Case belongs to the immutable Benchmark Version used by the Run. - For imported outputs, compare external identifiers with the immutable Case IDs shown by the mapping surface. - Confirm that the intended output column and candidate metadata were selected before import. ## Fix - For managed execution, correct the Harness or execution problem and start a new Run against the same Benchmark Version when that boundary is still intended. - For imported outputs, correct the external rows and repeat mapping with immutable Case IDs. - Keep managed and imported-output diagnoses separate; an imported response does not become a saved Harness Version. - Change the current Dataset and create a new Snapshot only when Case membership itself is wrong—not merely because a Run response is missing. ## Prevention - Save the exact Harness Version before managed execution and review execution settings. - Export or preserve immutable Case IDs through external generation. - Validate several mapped rows before creating an output-only Run. - Use Run Metadata to identify the candidate or external output set without pretending it is a Harness Version. ## Related task pages {% related-card-grid title="Related task pages" %} - [Import and prepare cases](/docs/operating-manual/import-and-prepare-cases) - [Run a benchmark](/docs/benchmark-evaluations/run-evaluation) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Target model outputs](/docs/object-model/outputs) - [Input-only and output-attached cases](/docs/assets/cases) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %} ## Source confidence Code-backed: the Run modal and Run detail establish managed execution identity and status; the output-mapping modal and List surface establish imported output-only Runs and unmatched Case handling. Input-only Cases are valid inputs to managed execution. --- id: object-model.outputs title: Outputs summary: Distinguish managed Run responses, imported output-only Runs, and Case-scoped reference outputs. kind: reference product_area: object_model status: stable updated: 2026-08-23 canonical: /docs/object-model/outputs --- # Outputs Understand the outputs Teammately reviews, compares, and turns into correctness evidence. ## Definition An output is response evidence associated with a Case, but its identity depends on the workflow that produced it. A managed Evaluation Run generates a response from a saved Harness Version. An imported output-only Run maps externally produced responses to immutable Case IDs. A Case may also expose a golden or reference response where the current surface supports it. Do not collapse these into one generic “accepted output” lifecycle. Imported outputs are not Harness Versions, Run responses are not automatically reference answers, and reviewer comments are not outputs. ## Fields, states, or lifecycle rules - A managed Run response is bound to the Run, saved Harness Version, Benchmark Version, and Case. - An imported output-only Run joins external rows to immutable Case IDs and retains imported candidate metadata. - Output-only Runs can be scored and inspected, but cannot be activated, optimized in Improve, or used as Harness columns in Compare or Arena. - A golden or reference output is Case-scoped example evidence where exposed; it does not become a global Policy. - Mapping errors can make evidence misleading even when the external response data is unchanged. - This page documents object boundaries, not model execution, provider behavior, or public output schemas. ## Related objects Read outputs with [Cases](/docs/object-model/cases), [Reference and golden outputs](/docs/object-model/reference-and-golden-outputs), [Run an evaluation](/docs/benchmark-evaluations/run-evaluation), and [Map external outputs](/docs/benchmark-evaluations/output-mapping). {% example-demo title="Target model outputs boundary" %} Case: A user asks whether a product is compatible with equipment they own. Candidate output: The assistant says the product will work. Interpretation: The benchmark failure belongs to the candidate output, but the reviewer still needs the case context and applicable rubric to explain why it failed. {% /example-demo %} ## Source confidence Code-backed: Benchmark Dataset output mapping and Evaluation Run detail distinguish imported output-only evidence from managed Run responses; review and Case-content sources preserve the Case boundary. Provider execution and customer serialization are outside this reference. ## Related task pages {% related-card-grid title="Related task pages" %} - [Run an evaluation](/docs/benchmark-evaluations/run-evaluation) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) - [Connect model outputs](/docs/integrations/connect-model-outputs) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} --- id: troubleshooting.output-mapping title: Output mapping troubleshooting summary: Fix external output rows that do not join cleanly to immutable Case IDs in an output-only Run. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/output-mapping --- # Output mapping troubleshooting Use this when externally produced responses exist but the mapping preview cannot join them cleanly to the Cases in the selected Benchmark Version. ## Symptom Rows are unmatched, duplicated, or mapped to the wrong Case ID, or the wrong column was selected as the response. No output-only Run should be trusted until the mapping preview is correct. ## Likely causes - External rows use row order or display text instead of immutable Case IDs. - The file contains Case IDs from another Benchmark Version or Dataset export. - The selected response column or identifier column is wrong. - Several candidate output sets were combined without distinct metadata. ## Diagnostic checks - Compare several affected case IDs with the output source IDs. - Check whether the output set has candidate metadata such as model, prompt, retrieval version, or run identity. - Confirm the source file was produced from the same immutable Case population. - Inspect duplicate, blank, and unknown identifier counts in the mapping preview. ## Fix - Correct the external file and map using immutable Case IDs rather than row position. - Select the intended response column and keep one candidate output set per output-only Run. - Create a new output-only Run from the corrected mapping. Existing Runs remain evidence under their original mapping. - Repair missing Case context through the Case workflow, not during output mapping; create a new version boundary when Case content changes. ## Prevention - Preserve source IDs through import and output generation. - Sample mapped outputs before expert contribution work and before Evaluation Runs. - Keep candidate identity in run metadata. - Treat each imported output-only Run as its own evidence record; it is not a saved Harness Version. ## Related task pages {% related-card-grid title="Related task pages" %} - [Importing cases](/docs/operating-manual/import-and-prepare-cases) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Target model outputs](/docs/object-model/outputs) - [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases) - [Benchmark Evaluations](/docs/benchmark-evaluations) {% /related-card-grid %} ## Source confidence Code-backed: the current output-mapping modal defines identifier and output-column selection, mapping preview, unmatched rows, metadata, and creation of an output-only Run. The Runs workspace keeps that imported evidence distinct from saved Harness Versions. --- id: assets.cases title: Cases summary: Understand canonical project cases, their input and materials, and how they become members of benchmark datasets. kind: reference product_area: assets status: stable updated: 2026-08-22 canonical: /docs/assets/cases --- # Cases ## Definition A Case is a project-level situation used for expert contribution, benchmark coverage, or candidate evaluation. It has canonical input content and may include declared supporting materials. Cases live in the Assets pool and can be selected into one or more benchmark datasets. The Project Input Schema determines how the primary input and materials are represented. A benchmark snapshot determines which selected cases belong to one versioned evidence boundary. ## Fields, states, or lifecycle rules - Canonical primary input is stored under `content.input`. - Optional supporting values or artifacts are stored under `content.case_materials` according to the project's declared keys. - `record_content.case_view` is a rendered projection used for inspection and delivery; it is not a second editable payload. - Inputs can use plain-text, chat, or structured architecture as configured by the project. - Materials can include admitted artifact families and must satisfy any required-field and file-extension rules. - A project Case is not automatically part of every benchmark. Benchmark Datasets owns selection and snapshots. - Generated or imported Cases should be reviewed for clarity, source authority, and schema conformance before they are trusted as benchmark evidence. - Static case materials and executable Worlds remain separate. A document supplied to a Harness does not become a world merely because it affects execution. ## Case identity and change Treat the persisted case identity as opaque. Do not construct IDs in client code or documentation. When case content changes materially, benchmark interpretation must use a snapshot or version boundary that makes the selected content clear. Responses produced by a Harness are evaluation outputs attached to a Run. They are not the primary case input. Expert-authored acceptable examples can inform standards or contribution work, but the current evaluation contract should remain explicit about which candidate produced each response. {% example-demo title="Example: multimaterial case" %} A chat case asks whether an exception applies. Its required `current_policy` PDF and optional `account_history` table are stored as case materials admitted by Project Input Schema. The rendered case view presents the conversation and both materials. A benchmark snapshot selects the case, and a Run records the evaluated Harness response separately. {% /example-demo %} ## Source confidence Code-backed: the active Assets Cases route and case-content services define canonical input, case materials, and the rendered case view. Public import or export APIs are outside this reference unless separately documented. ## Related task pages {% related-card-grid title="Related task pages" %} - [Configure Project Input Schema](/docs/project-settings/input-schema) - [Work with Benchmark Datasets](/docs/benchmark-datasets) - [Manage benchmark coverage](/docs/coverage-management) {% /related-card-grid %}