Teammately Docs
Docs menu

error

Missing Outputs

Separate a missing managed Run response from an unmatched row in an imported output-only Run.

Missing Outputs

Use this when an Evaluation Run has no response for an intended Case. First identify whether the evidence came from managed Harness execution or an imported output-only Run.

Symptom

A managed Run is incomplete or has a failed or unevaluable Case attempt, or an imported output-only Run shows an unmatched Case row. An input-only Case by itself is not an error: managed execution is expected to generate its response.

Likely causes

  • Managed execution failed, stopped, or could not evaluate the Case.
  • The wrong saved Harness Version or execution settings were selected.
  • An external output row has a missing, unknown, or duplicate immutable Case ID.
  • The imported file omitted Cases that the output-only Run was intended to cover.

Diagnostic checks

  • Open the Run and check its lifecycle, attempts, failed Cases, and selected Harness Version.
  • Confirm that the Case belongs to the immutable Benchmark Version used by the Run.
  • For imported outputs, compare external identifiers with the immutable Case IDs shown by the mapping surface.
  • Confirm that the intended output column and candidate metadata were selected before import.

Fix

  • For managed execution, correct the Harness or execution problem and start a new Run against the same Benchmark Version when that boundary is still intended.
  • For imported outputs, correct the external rows and repeat mapping with immutable Case IDs.
  • Keep managed and imported-output diagnoses separate; an imported response does not become a saved Harness Version.
  • Change the current Dataset and create a new Snapshot only when Case membership itself is wrong—not merely because a Run response is missing.

Prevention

  • Save the exact Harness Version before managed execution and review execution settings.
  • Export or preserve immutable Case IDs through external generation.
  • Validate several mapped rows before creating an output-only Run.
  • Use Run Metadata to identify the candidate or external output set without pretending it is a Harness Version.

Related task pages

Task / Operating ManualImport and prepare casesBring cases into the project, conform them to Project Input Schema, inspect materials, and prepare benchmark selection.Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.Task / Benchmark EvaluationsMap External Evaluation OutputsImport reference outputs, map them to immutable benchmark Cases and attempts, and inspect the resulting output-only Run.

Related reference pages

Reference / Object ModelOutputsDistinguish managed Run responses, imported output-only Runs, and Case-scoped reference outputs.Reference / AssetsCasesUnderstand canonical project cases, their input and materials, and how they become members of benchmark datasets.Reference / Object ModelBenchmarksUnderstand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.

Source confidence

Code-backed: the Run modal and Run detail establish managed execution identity and status; the output-mapping modal and List surface establish imported output-only Runs and unmatched Case handling. Input-only Cases are valid inputs to managed execution.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Object Model Outputs Distinguish managed Run responses, imported output-only Runs, and Case-scoped reference outputs. Recovery Library Output mapping troubleshooting Fix external output rows that do not join cleanly to immutable Case IDs in an output-only Run. Assets Cases Understand canonical project cases, their input and materials, and how they become members of benchmark datasets.
AI context