Teammately Docs
Docs menu

error

Unbalanced Coverage

Fix benchmarks that overrepresent easy or common cases while missing important behavior.

Unbalanced Coverage

Use this when benchmark evidence overrepresents some behavior and misses other behavior that matters before human review.

Symptom

A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent.

Likely causes

  • Case import mirrored production volume instead of product risk.
  • Dimensions or metadata do not represent the behavior areas that matter.
  • Synthetic cases were added in one area but not reviewed across the full risk boundary.
  • Coverage was not refreshed after new signals, policy changes, or failures.

Diagnostic checks

  • Inspect Benchmark Datasets → Representation and the Coverage Management overview by the configured Coverage Facets.
  • Look for policies or rubrics with very few applicable cases.
  • Compare production or expert signals with benchmark case distribution.
  • Inspect whether new cases are candidate-only or actually in use.

Fix

  • Add representative, boundary, or adversarial cases for undercovered areas.
  • Refresh dimensions or metadata before judging balance.
  • Review and promote candidate cases into the benchmark only after context is complete.
  • Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance.

Prevention

  • Define coverage targets before benchmark runs.
  • Track candidate and in-use cases separately.
  • Review coverage after policy, rubric, or product behavior changes.
  • Do not use aggregate scores without a coverage note.

Related task pages

Task / Benchmark DatasetsDataset RepresentationAnalyze how distinct benchmark Cases are distributed across facets, evaluator rules, and provenance.Task / Coverage ManagementSet Up Benchmark CoverageDefine benchmark intent, facet handling, artifact preferences, and evidence requirements before generating coverage work.Task / Coverage ManagementCoverage StoriesOrganize benchmark coverage intent into governed Stories and testable facet tuples.

Related reference pages

Reference / Object ModelCase Pool surfaceUnderstand Case Pool as the Assets view for reusable project Cases, candidates, and explicit Benchmark selection.Reference / Object ModelCoverage dimensionsOrganize cases by the behavior axes that matter to product correctness.Reference / Object ModelBenchmarksUnderstand a Benchmark as the durable program that owns benchmark-scoped coverage, evidence boundaries, evaluations, and improvement work.

Source confidence

Code-backed: Benchmark Dataset Representation, Coverage Management, Get Started, and Coverage Stories expose the current selected distribution, benchmark denominator, guidance, and named gaps used for this diagnosis.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Coverage Gaps Find missing or underrepresented behavior areas before benchmark evidence becomes misleading. Coverage Engineering Case Segmentation Segment cases into meaningful behavior groups so benchmark coverage is explainable. Coverage Engineering Plan Benchmark Coverage Apply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work.
AI context