Teammately Docs
Docs menu

task

Inspect Evaluation Results

Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.

Inspect Evaluation Results

Prerequisites

  • A visible Run or Run Group with output or evaluation progress.
  • Access to the exact Benchmark and Harness Version evidence.

Start with completeness and identity, then move from aggregate signals to the Cases and evaluator failures that support them.

Steps

  1. Open Dashboard and confirm the Benchmark Version, candidate Harness Version, Run Group type, attempt count, and evaluation progress.
  2. Read rankings with their metric family and uncertainty. Distinguish average score, passed at least once, and passed every time over observed Runs. Inspect Run counts and coverage; historical group-specific pass@n and pass^n retain their original meanings.
  3. Open List → Runs to inspect group and Run status, output progress, evaluation progress, metadata, and available resource telemetry.
  4. Open List → Evaluation results for the Case summary, outcome, failed Policies, failed Rubrics, and evaluated count.
  5. Use Arena for pairwise disagreement or Compare for a Harness matrix across Cases, evaluator facts, or Coverage Facets.
  6. Classify the next action as candidate work, evaluator clarification, Case correction, coverage work, external-output remapping, or no action.

The List results surface is intentionally compact. Do not claim that it exposes full execution trajectories. The Traces / Spans segment currently reports a capability fence because the benchmark API does not provide evaluation execution traces.

Reading incomplete and repeated evidence

An aggregate calculated over fewer evaluable Cases can look better while covering less evidence. Record evaluated, incomplete, and missing counts before comparing candidates. For repeated groups, inspect whether the configured number of attempts exists for every candidate and whether one failed attempt changes the metric interpretation.

Cost, tokens, and latency help route operational work but are nullable telemetry. Missing capture means unknown, not free or instantaneous execution.

Worked example

Example: apparent gain from incomplete evidence

Harness B leads the overall table, but List shows that twelve difficult Cases are still unevaluated for B. Arena also reports incomplete pairs. The operator waits for terminal evidence instead of starting Improve from a ranking that covers a smaller Case population.

Object and state changes

Inspection, filtering, and navigation are read-only. Starting Improve, a Contribution, coverage work, or a later Run creates separate durable work while preserving the inspected evidence.

Success criteria

  • Identity, completeness, metric family, and uncertainty are explicit.
  • Important signals resolve to Cases and admitted evaluator outcomes.
  • The next action targets the responsible artifact or candidate boundary.

Common failure modes

  • Reporting rank without the evaluated population.
  • Inventing execution traces from the unavailable segment.
  • Starting candidate work when the Case or Rubric is wrong.

Related reference pages

Concept / Benchmark EvaluationsBenchmark EvaluationsRun and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare.Task / Benchmark EvaluationsArena and RankingsInterpret pairwise candidate disagreement, governed metric families, repeated-sampling ranks, and uncertainty.Task / Benchmark DatasetsDataset SnapshotsFreeze Cases, evaluator links, and representation facts as an immutable benchmark evidence boundary.

Related troubleshooting pages

Troubleshooting / Recovery LibraryBenchmark Results Changed UnexpectedlyDiagnose result changes across target behavior, benchmark cases, standards, and versions.Troubleshooting / Recovery LibraryBenchmark run troubleshootingDiagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely.Troubleshooting / Recovery LibraryMissing OutputsSeparate a missing managed Run response from an unmatched row in an imported output-only Run.

Source confidence

Code-backed: Dashboard, List, Run detail, and workspace types establish result summaries, completion, rankings, repeated metrics, telemetry, and the current trace capability fence.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Benchmark Evaluations Benchmark Evaluations Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare. Benchmark Evaluations Arena and Rankings Interpret pairwise candidate disagreement, governed metric families, repeated-sampling ranks, and uncertainty. Benchmark Evaluations Compare Harness Versions Compare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.
AI context