Teammately Docs
Docs menu

recipe

Finding Coverage Gaps Before Review

Use coverage dimensions, failures, and expert signals to decide where evidence is incomplete.

Finding Coverage Gaps Before Review

Use this playbook when benchmark evidence looks plausible overall but the team needs to know whether important behavior slices are missing or underrepresented.

Review-readiness trigger

Use this before a customer relies on Benchmark evidence for human review, especially when results are dominated by common Cases, a critical Rubric has few applicable Cases, or specialists identify behavior that the Dataset does not represent.

Coverage audit

  1. Confirm the exact Benchmark Version and Run completeness before interpreting its distribution.
  2. Open Benchmark Datasets → Representation and inspect the selected Dataset across the configured Coverage Facets.
  3. Compare thin or empty slices with Coverage Stories, failed Cases, specialist observations, and must-level Rubrics with few applicable Cases.
  4. Classify each issue: missing vocabulary, missing Case, unreviewed candidate, missing Dataset membership, or unclear correctness standard.
  5. Route vocabulary changes to Coverage Facets, unclear standards to Expert Contributions, and missing Cases to Assets Synthesis or Case Foundry.
  6. Review candidates in Case Review, select the intended Cases in Benchmark Datasets, and create a new Snapshot.
  7. Run the relevant saved Harness Versions against the new Benchmark Version and state any still-unrepresented risk in the customer's review context.

Evidence threshold

Coverage is ready when critical slices are named, their selected Case counts are visible, candidate-only Cases are not counted as evidence, and remaining gaps are explicit. A balanced-looking aggregate count is not sufficient.

Worked example

Compatibility gaps

Dataset Representation shows many ordinary recommendations but only two accessory-compatibility Cases and no discontinued-model Cases. A Coverage Story records the gap. Case Foundry prepares cross-brand adapter, ambiguous model-number, and discontinued-model candidates; specialists review the source conditions in Case Review. Only the admitted Cases enter the new Snapshot, and the customer withholds compatibility conclusions until the rerun is complete.

Evidence to collect

  • Benchmark Version, Snapshot, selected Case count, and Dataset Representation.
  • Coverage Stories and specialist observations that identify consequential gaps.
  • Candidate lineage, Case Review decisions, and final Dataset membership.
  • New Snapshot and Benchmark Version after membership changes.
  • Rerun completeness and an explicit list of remaining unsupported conclusions.

Related docs

Task / Benchmark DatasetsDataset RepresentationAnalyze how distinct benchmark Cases are distributed across facets, evaluator rules, and provenance.Task / Coverage ManagementCoverage StoriesOrganize benchmark coverage intent into governed Stories and testable facet tuples.Task / Coverage ManagementCase ReviewPrepare, inspect, refine, and admit Case candidates and generated materials into the benchmark dataset.Task / Benchmark EvaluationsInspect Evaluation ResultsTrace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.Task / Operating ManualImport and prepare casesBring cases into the project, conform them to Project Input Schema, inspect materials, and prepare benchmark selection.

Source confidence

Doctrine-backed: the approved product model requires deliberate coverage before Benchmark evidence is trusted. Linked code-backed pages define current Representation, Coverage Story, Case Review, Snapshot, and Run boundaries.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Coverage Gaps Find missing or underrepresented behavior areas before benchmark evidence becomes misleading. Coverage Engineering Plan Benchmark Coverage Apply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work. Benchmark Evaluations Inspect Evaluation Results Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.
AI context