--- id: coverage.coverage-gaps title: Coverage Gaps summary: Find missing or underrepresented behavior areas before benchmark evidence becomes misleading. kind: task product_area: coverage_engineering status: stable updated: 2026-08-23 canonical: /docs/coverage-engineering/coverage-gaps --- # Coverage Gaps ## When to use it Use this task when the team suspects that a benchmark result is incomplete because the case set does not represent an important behavior area. A coverage gap is not just a low score. It is a missing or thin slice of the behavior space: a dimension value, ontology category, source condition, user intent, boundary scenario, policy exception, failure cluster, or product flow that should be represented before results are trusted. Coverage gaps matter because Teammately helps the team reason about whether the benchmark actually represents the correctness space, instead of only running checks over available examples. ## Prerequisites - A Benchmark Dataset or a named intended behavior slice already exists. - Dimensions or ontology values are available, or the team knows which behavior axis is missing. - Relevant Evaluation Runs, failure clusters, Expert Contribution findings, or product signals are available for inspection. - Policies and rubrics are clear enough that the team can tell whether the problem is missing coverage rather than weak standards. ## Required role or permission AI engineers, evaluation owners, and product owners usually identify coverage gaps together. Experts may be needed when the missing behavior depends on domain judgment. If the UI blocks investigation or case changes, inspect project membership and artifact access before changing the benchmark. ### Task steps: Coverage Gaps 1. Name the behavior area that may be missing: dimension, ontology value, product flow, policy exception, source condition, or boundary scenario. 2. Inspect **Benchmark Datasets → Representation** and the current Benchmark Version. Check whether the area is absent, represented by too few selected Cases, or represented only by easy examples. 3. Compare the suspected gap against evaluation failures, failure clusters, Expert Contribution notes, and recent product signals. 4. Rule out look-alike problems: missing outputs, stale cases, weak applicability logic, overly broad policies, ambiguous rubrics, or output mapping errors. 5. Route the gap: update Coverage Facets, create a Coverage Story, source or synthesize Cases, request an Expert Contribution, or select already reviewed Cases in Benchmark Datasets. 6. Review candidates in Case Review, create a new Snapshot when membership changes, and preserve the gap rationale in the owning coverage surfaces. ![Case Pool table with selected cases and an action bar for adding cases to a benchmark.](/docs-assets/assets/screenshots/case-pool-selected-action-bar.png) When a gap points to specific candidates, the operator can select cases and prepare them for benchmark membership. ## Object and state changes Confirming a gap can create a Coverage Story, candidate Cases, Coverage Facet changes, Case Review work, selected Dataset changes, or an Expert Contribution. A gap does not silently change historical Benchmark meaning. When selected membership changes, create a new Dataset Snapshot and Benchmark Version before treating the revised set as reproducible evidence. ## Success criteria - The missing or underrepresented behavior area is named precisely. - The team can explain why the issue is a true coverage gap rather than missing outputs, weak applicability, stale artifacts, or mapping errors. - The resulting case, dimension, ontology, or benchmark change is traceable to source evidence or expert judgment. - Future benchmark results can distinguish behavior improvement from coverage refresh. ## Common failure modes - Treating a model failure as a coverage gap when the benchmark already contains representative cases. - Adding many similar cases without naming the missing dimension or ontology value. - Refreshing benchmark coverage without preserving the version boundary. - Mistaking missing outputs or output mapping failures for missing coverage. - Creating synthetic cases that are unrealistic because they lack source context or expert judgment. - Ignoring a small high-risk slice because aggregate coverage looks balanced. {% example-demo title="Boundary case for enterprise search" %} Raw case: An employee asks for a policy that changed last week, and the retrieved documents contain both old and new guidance. Expert judgment: Coverage must include cases where stale and current sources conflict. Policy: Answers must prefer the approved current source and disclose conflicts when confidence is low. Applicability: Applies when retrieval includes multiple policy versions or stale documents. Binary rubric: The answer identifies the current source or asks for confirmation instead of blending policies. Benchmark result: A candidate output fails because it combines old and new terms into one invented policy. Interpretation: Coverage notes show whether stale-source boundary behavior is represented before the next run is trusted. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Candidate and In-Use Cases](/docs/coverage-engineering/candidate-and-in-use-cases) - [Case pool](/docs/coverage-engineering/case-pool) - [Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) - [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots) - [Case versions](/docs/governance/case-versioning) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Access troubleshooting](/docs/troubleshooting/authentication) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Weak applicability logic](/docs/troubleshooting/weak-applicability-logic) - [Missing outputs](/docs/troubleshooting/missing-outputs) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} ## Source confidence Code-backed: Benchmark Dataset Representation exposes selected distribution; Coverage Management and Coverage Stories expose benchmark needs; Case Review exposes the admission boundary for prepared Cases. Human judgment determines whether an observed thin slice is consequential.