# Unbalanced Coverage Generated: 2026-09-13T04:39:11.023Z Source build: local Canonical docs: https://teammately.ai/docs --- id: troubleshooting.unbalanced-coverage title: Unbalanced Coverage summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/unbalanced-coverage --- # Unbalanced Coverage Use this when benchmark evidence overrepresents some behavior and misses other behavior that matters before human review. ## Symptom A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent. ## Likely causes - Case import mirrored production volume instead of product risk. - Dimensions or metadata do not represent the behavior areas that matter. - Synthetic cases were added in one area but not reviewed across the full risk boundary. - Coverage was not refreshed after new signals, policy changes, or failures. ## Diagnostic checks - Inspect **Benchmark Datasets → Representation** and the Coverage Management overview by the configured Coverage Facets. - Look for policies or rubrics with very few applicable cases. - Compare production or expert signals with benchmark case distribution. - Inspect whether new cases are candidate-only or actually in use. ## Fix - Add representative, boundary, or adversarial cases for undercovered areas. - Refresh dimensions or metadata before judging balance. - Review and promote candidate cases into the benchmark only after context is complete. - Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance. ## Prevention - Define coverage targets before benchmark runs. - Track candidate and in-use cases separately. - Review coverage after policy, rubric, or product behavior changes. - Do not use aggregate scores without a coverage note. ## Related task pages {% related-card-grid title="Related task pages" %} - [Inspect Dataset representation](/docs/benchmark-datasets/representation) - [Configure Coverage Management](/docs/coverage-management/get-started) - [Work with Coverage Stories](/docs/coverage-management/coverage-stories) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Case pool](/docs/object-model/case-pool) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %} ## Source confidence Code-backed: Benchmark Dataset Representation, Coverage Management, Get Started, and Coverage Stories expose the current selected distribution, benchmark denominator, guidance, and named gaps used for this diagnosis. --- id: coverage.coverage-gaps title: Coverage Gaps summary: Find missing or underrepresented behavior areas before benchmark evidence becomes misleading. kind: task product_area: coverage_engineering status: stable updated: 2026-08-23 canonical: /docs/coverage-engineering/coverage-gaps --- # Coverage Gaps ## When to use it Use this task when the team suspects that a benchmark result is incomplete because the case set does not represent an important behavior area. A coverage gap is not just a low score. It is a missing or thin slice of the behavior space: a dimension value, ontology category, source condition, user intent, boundary scenario, policy exception, failure cluster, or product flow that should be represented before results are trusted. Coverage gaps matter because Teammately helps the team reason about whether the benchmark actually represents the correctness space, instead of only running checks over available examples. ## Prerequisites - A Benchmark Dataset or a named intended behavior slice already exists. - Dimensions or ontology values are available, or the team knows which behavior axis is missing. - Relevant Evaluation Runs, failure clusters, Expert Contribution findings, or product signals are available for inspection. - Policies and rubrics are clear enough that the team can tell whether the problem is missing coverage rather than weak standards. ## Required role or permission AI engineers, evaluation owners, and product owners usually identify coverage gaps together. Experts may be needed when the missing behavior depends on domain judgment. If the UI blocks investigation or case changes, inspect project membership and artifact access before changing the benchmark. ### Task steps: Coverage Gaps 1. Name the behavior area that may be missing: dimension, ontology value, product flow, policy exception, source condition, or boundary scenario. 2. Inspect **Benchmark Datasets → Representation** and the current Benchmark Version. Check whether the area is absent, represented by too few selected Cases, or represented only by easy examples. 3. Compare the suspected gap against evaluation failures, failure clusters, Expert Contribution notes, and recent product signals. 4. Rule out look-alike problems: missing outputs, stale cases, weak applicability logic, overly broad policies, ambiguous rubrics, or output mapping errors. 5. Route the gap: update Coverage Facets, create a Coverage Story, source or synthesize Cases, request an Expert Contribution, or select already reviewed Cases in Benchmark Datasets. 6. Review candidates in Case Review, create a new Snapshot when membership changes, and preserve the gap rationale in the owning coverage surfaces. ![Case Pool table with selected cases and an action bar for adding cases to a benchmark.](/docs-assets/assets/screenshots/case-pool-selected-action-bar.png) When a gap points to specific candidates, the operator can select cases and prepare them for benchmark membership. ## Object and state changes Confirming a gap can create a Coverage Story, candidate Cases, Coverage Facet changes, Case Review work, selected Dataset changes, or an Expert Contribution. A gap does not silently change historical Benchmark meaning. When selected membership changes, create a new Dataset Snapshot and Benchmark Version before treating the revised set as reproducible evidence. ## Success criteria - The missing or underrepresented behavior area is named precisely. - The team can explain why the issue is a true coverage gap rather than missing outputs, weak applicability, stale artifacts, or mapping errors. - The resulting case, dimension, ontology, or benchmark change is traceable to source evidence or expert judgment. - Future benchmark results can distinguish behavior improvement from coverage refresh. ## Common failure modes - Treating a model failure as a coverage gap when the benchmark already contains representative cases. - Adding many similar cases without naming the missing dimension or ontology value. - Refreshing benchmark coverage without preserving the version boundary. - Mistaking missing outputs or output mapping failures for missing coverage. - Creating synthetic cases that are unrealistic because they lack source context or expert judgment. - Ignoring a small high-risk slice because aggregate coverage looks balanced. {% example-demo title="Boundary case for enterprise search" %} Raw case: An employee asks for a policy that changed last week, and the retrieved documents contain both old and new guidance. Expert judgment: Coverage must include cases where stale and current sources conflict. Policy: Answers must prefer the approved current source and disclose conflicts when confidence is low. Applicability: Applies when retrieval includes multiple policy versions or stale documents. Binary rubric: The answer identifies the current source or asks for confirmation instead of blending policies. Benchmark result: A candidate output fails because it combines old and new terms into one invented policy. Interpretation: Coverage notes show whether stale-source boundary behavior is represented before the next run is trusted. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Candidate and In-Use Cases](/docs/coverage-engineering/candidate-and-in-use-cases) - [Case pool](/docs/coverage-engineering/case-pool) - [Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) - [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots) - [Case versions](/docs/governance/case-versioning) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Access troubleshooting](/docs/troubleshooting/authentication) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Weak applicability logic](/docs/troubleshooting/weak-applicability-logic) - [Missing outputs](/docs/troubleshooting/missing-outputs) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} ## Source confidence Code-backed: Benchmark Dataset Representation exposes selected distribution; Coverage Management and Coverage Stories expose benchmark needs; Case Review exposes the admission boundary for prepared Cases. Human judgment determines whether an observed thin slice is consequential. --- id: coverage.case-segmentation title: Case Segmentation summary: Segment cases into meaningful behavior groups so benchmark coverage is explainable. kind: concept product_area: coverage_engineering status: stable updated: 2026-08-23 canonical: /docs/coverage-engineering/case-segmentation --- # Case Segmentation ## Definition Case segmentation is the act of reading a Case population by meaningful behavior slices instead of only as a total count. In Teammately, reusable Dimensions and ontology values provide governed segmentation axes; Project Topics and Case Construction Patterns provide additional intent; Benchmark Dataset Representation shows how selected Cases occupy those axes. A segment is an analytical grouping, not a new Case object or a hidden approval state. ## Why it matters Aggregate benchmark results can hide concentrated success and consequential absence. A candidate may score well overall while failing every Case with missing evidence, a particular jurisdiction, or a high-impact exception. Segmentation makes those differences inspectable and gives Coverage Stories a concrete population to address. ## Where it appears in the product Define reusable axes in **Coverage Facets → Dimensions & Ontology**. Classify Cases against those values through supported Case and coverage workflows. Use **Benchmark Datasets → Representation** to inspect the current selected population. Use Coverage Management when a sparse or absent intersection needs a Story, Case Foundry work, or an Expert Contribution. ## Artifacts it affects Keep three identities separate: the reusable Dimension, the ontology value assigned to a Case, and the Benchmark Dataset membership being analyzed. Changing a value can alter current Representation without modifying an older Snapshot. Adding a Case can change counts without changing the segmentation model. A filter is a view over evidence, not a new evidence object. ## Operational check Choose axes that describe material behavior, context, risk, or source conditions. Check distinct Case counts as well as assignments, inspect unclassified and conflicting values, and avoid interpreting overlapping segments as additive totals. When reporting a result by segment, name the Benchmark Version whose Case assignments support the analysis. {% example-demo title="Finding a hidden source-authority gap" %} The dataset contains many high-impact Cases and many Cases with superseded sources, but Representation shows almost no Cases where both conditions occur together. The team creates a Coverage Story for that intersection. Case Foundry prepares candidates, reviewers confirm their realism, and a later Snapshot records the expanded segment. The Dimension definitions remain reusable across other Benchmarks. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Coverage Gaps](/docs/coverage-engineering/coverage-gaps) - [Product quickstart](/docs/quickstart) - [Task index](/docs/operating-manual/task-index) {% /related-card-grid %} ## Source confidence Code-backed: the Representation explorer and Dimension classification types establish facet-based Case grouping, assigned and missing values, and benchmark-scoped distribution views. Coverage significance and gap priority remain accountable human judgments. --- id: coverage.plan-benchmark-coverage title: Plan Benchmark Coverage summary: Apply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work. kind: task product_area: coverage_engineering status: stable updated: 2026-09-07 canonical: /docs/coverage-engineering/plan-benchmark-coverage --- # Plan Benchmark Coverage Plan coverage by applying reusable project facets to one benchmark and comparing the intended behavior space with the selected dataset representation. ## Prerequisites - A selected project and benchmark. - A clear benchmark purpose. - Relevant Dimensions, Project Topics, and Case Construction Patterns, or enough project knowledge to create them. - Existing Cases or a plan for sourcing and constructing them. ## Steps 1. Review **Coverage Facets** at project scope. Confirm that Dimensions, Project Topics, and Case Construction Patterns describe reusable behavior structure rather than one benchmark's current case count. 2. Open the benchmark and select **Coverage Management → Get Started**. 3. Define the benchmark-specific coverage guidance and confirm setup readiness. 4. Open Coverage Management and inspect current dataset representation across the relevant facets and tuples. 5. Name important thin or absent combinations as Coverage Stories. Explain why each slice matters and what evidence would make it usable. 6. Route the gap according to its cause: Case Foundry or case sourcing for missing situations, Expert Contributions for missing judgment, Correctness Governance for missing standards, or Benchmark Datasets for missing selection. 7. Review generated or contributed cases in Case Review before relying on them. 8. Update dataset selection and create a new snapshot when the represented evidence changes materially. ## Object and state changes This task can update benchmark coverage setup, representation guidance, Coverage Stories, Case Foundry work, case-review state, contribution requests, dataset selection, and snapshots. Project Coverage Facets may also change when the work discovers a reusable missing axis or construction pattern. ## Success criteria - The benchmark purpose maps to explicit project Coverage Facets. - Important combinations have selected evidence or a named gap. - Each gap is routed to a responsible artifact or workstream. - Constructed cases pass case review and Project Input Schema checks. - Dataset snapshots make material coverage changes explicit. ## Common failure modes - Using case count as the coverage goal. - Creating benchmark-only tags where a reusable Dimension or Topic is needed. - Treating response-variation guidance as coverage structure. - Generating cases before defining which gap they should close. - Trusting representation after selection changes without a new snapshot boundary. {% example-demo title="Example: plan high-impact exception coverage" %} The team maps exception type, source authority, and customer impact. Representation shows many low-impact ordinary cases but no high-impact cases with conflicting authority. A Coverage Story names the gap, an expert Contribution clarifies the controlling rule, and Case Foundry prepares cases for the missing tuple before a new snapshot is created. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Engineering](/docs/coverage-engineering) - [Coverage Management](/docs/coverage-management) - [Benchmark Datasets](/docs/benchmark-datasets) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Stale dimensions](/docs/troubleshooting/stale-dimensions) - [Unrealistic synthetic cases](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} ## Source confidence Code-backed: current setup, overview, representation, Coverage Story, Case Foundry, and Case Review routes support this workflow.