Teammately Docs
Docs menu

task

Plan Benchmark Coverage

Apply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work.

Plan Benchmark Coverage

Plan coverage by applying reusable project facets to one benchmark and comparing the intended behavior space with the selected dataset representation.

Prerequisites

  • A selected project and benchmark.
  • A clear benchmark purpose.
  • Relevant Dimensions, Project Topics, and Case Construction Patterns, or enough project knowledge to create them.
  • Existing Cases or a plan for sourcing and constructing them.

Steps

  1. Review Coverage Facets at project scope. Confirm that Dimensions, Project Topics, and Case Construction Patterns describe reusable behavior structure rather than one benchmark's current case count.
  2. Open the benchmark and select Coverage Management → Get Started.
  3. Define the benchmark-specific coverage guidance and confirm setup readiness.
  4. Open Coverage Management and inspect current dataset representation across the relevant facets and tuples.
  5. Name important thin or absent combinations as Coverage Stories. Explain why each slice matters and what evidence would make it usable.
  6. Route the gap according to its cause: Case Foundry or case sourcing for missing situations, Expert Contributions for missing judgment, Correctness Governance for missing standards, or Benchmark Datasets for missing selection.
  7. Review generated or contributed cases in Case Review before relying on them.
  8. Update dataset selection and create a new snapshot when the represented evidence changes materially.

Object and state changes

This task can update benchmark coverage setup, representation guidance, Coverage Stories, Case Foundry work, case-review state, contribution requests, dataset selection, and snapshots. Project Coverage Facets may also change when the work discovers a reusable missing axis or construction pattern.

Success criteria

  • The benchmark purpose maps to explicit project Coverage Facets.
  • Important combinations have selected evidence or a named gap.
  • Each gap is routed to a responsible artifact or workstream.
  • Constructed cases pass case review and Project Input Schema checks.
  • Dataset snapshots make material coverage changes explicit.

Common failure modes

  • Using case count as the coverage goal.
  • Creating benchmark-only tags where a reusable Dimension or Topic is needed.
  • Treating response-variation guidance as coverage structure.
  • Generating cases before defining which gap they should close.
  • Trusting representation after selection changes without a new snapshot boundary.

Worked example

Example: plan high-impact exception coverage

The team maps exception type, source authority, and customer impact. Representation shows many low-impact ordinary cases but no high-impact cases with conflicting authority. A Coverage Story names the gap, an expert Contribution clarifies the controlling rule, and Case Foundry prepares cases for the missing tuple before a new snapshot is created.

Related reference pages

Concept / Coverage EngineeringCoverage EngineeringDesign the behavior space a benchmark must represent and connect reusable project facets to benchmark coverage work.Concept / Coverage ManagementCoverage ManagementManage benchmark coverage from setup through representation, Coverage Stories, case review, Case Foundry, and contribution requests.Concept / Benchmark DatasetsBenchmark DatasetsSelect benchmark Cases, inspect representation, and freeze immutable Snapshots for reproducible evidence.

Related troubleshooting pages

Troubleshooting / Recovery LibraryUnbalanced CoverageFix benchmarks that overrepresent easy or common cases while missing important behavior.Troubleshooting / Recovery LibraryStale DimensionsRefresh Dimensions and ontology values that no longer explain the current behavior space without rewriting historical evidence.Troubleshooting / Recovery LibrarySynthetic Cases That Feel UnrealisticImprove generated candidates whose surface details or behavior assumptions do not represent plausible target-system use.

Source confidence

Code-backed: current setup, overview, representation, Coverage Story, Case Foundry, and Case Review routes support this workflow.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Coverage Engineering Design the behavior space a benchmark must represent and connect reusable project facets to benchmark coverage work. Coverage Management Coverage Management Manage benchmark coverage from setup through representation, Coverage Stories, case review, Case Foundry, and contribution requests. Benchmark Datasets Benchmark Datasets Select benchmark Cases, inspect representation, and freeze immutable Snapshots for reproducible evidence.
AI context