Teammately Docs
Docs menu

task

Refresh coverage after product change

Reconcile coverage facets, Cases, benchmark membership, and Snapshots after the target system or its evidence changes.

Refresh coverage after product change

When to use it

Refresh coverage when new Cases reveal an unrepresented behavior, source material or product behavior changes, experts qualify an earlier assumption, or a governed Policy changes which situations matter. This is a coordinated workflow across Coverage Engineering—not a single refresh action.

Prerequisites

  • Name the changed signal and the date or version at which it changed.
  • Identify the Benchmark whose claims may be affected.
  • Preserve the current Snapshot and historical Runs; do not edit them to resemble the new state.
  • Decide who can confirm the changed behavior and who owns the resulting Benchmark Version.

Refresh benchmark coverage

  1. Open the Benchmark's Coverage Management overview and identify which coverage claim is no longer supported.
  2. Review Coverage Facets. Update Dimensions, ontology values, Project Topics, or Case Construction Patterns only when the behavior model itself changed.
  3. Return to the Case Pool. Source, upload, draft, or synthesize candidate Cases for the missing or changed region.
  4. Inspect the candidates for source context, realistic inputs, duplication, and the intended coverage labels. Keep uncertain Cases out of benchmark use.
  5. Use Coverage Management → Get Started and the overview to update coverage guidance. Use Case Review and Benchmark Datasets to change selected Cases deliberately.
  6. Create a new Dataset Snapshot and Benchmark Version for the revised evidence boundary.
  7. Run a new evaluation when current candidate evidence is required. Compare it with older Runs using the named Benchmark Versions.

Object and state changes

A refresh may change coverage-facet definitions, Case classifications, candidate Cases, selected benchmark Cases, and the next Snapshot. It does not rewrite an earlier Snapshot or make its Runs invalid. Older results remain evidence for their original version; the new version answers the current coverage question.

If only candidate behavior changed, keep the Benchmark Version fixed and run the new candidate against it. If the Case set, applicable standards, or coverage boundary changed, create a new Benchmark Version before interpreting a new Run as comparable.

Success criteria

  • The changed product reality maps to an explicit coverage facet or documented boundary.
  • Candidate Cases have enough source context to be reviewed and are not mistaken for in-use benchmark evidence.
  • The new selected set addresses the gap without silently removing still-important behavior.
  • The new Snapshot names a reproducible evidence boundary.
  • Comparisons distinguish candidate changes from Benchmark Version changes.

Common failure modes

  • Treating refresh as a single button and missing a changed facet, Case set, or Snapshot boundary.
  • Rewriting a historical Snapshot instead of creating a new one.
  • Adding generated or newly sourced Cases to a Benchmark before review.
  • Comparing Runs without naming whether the candidate, Benchmark Version, or both changed.

Worked example

A newly supported exception

A support assistant gains an approved exception path for one account tier. The team adds or revises the account-tier ontology, sources Cases for eligible and ineligible requests, reviews them in Case Review, updates the coverage guidance, and changes the selected Benchmark Dataset. A new Snapshot freezes the revised membership. Previous Runs still describe the old rule; new Runs evaluate the approved exception boundary.

Source confidence

Code-backed: Coverage Management, Coverage Facets, the Case Pool, Case Review, Benchmark Datasets, and Dataset Snapshots establish the current sequence and the objects that can change. The decision that a product change requires a refresh remains a team-owned interpretation of evidence.

Continue the workflow

Task / Coverage EngineeringCoverage GapsFind missing or underrepresented behavior areas before benchmark evidence becomes misleading.Reference / Coverage EngineeringDimensions and OntologyDefine reusable behavior axes and their allowed values, then inspect how cases and benchmarks cover them.Concept / Coverage EngineeringCase PoolUse the Case Pool to collect, triage, enrich, and promote candidate cases.Task / Coverage EngineeringPlan Benchmark CoverageApply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work.Concept / Coverage EngineeringBenchmark snapshotsFreeze a benchmark into a version so every run measures the same cases and judgment rules.

Diagnose refresh problems

Troubleshooting / Recovery LibraryStale DimensionsRefresh Dimensions and ontology values that no longer explain the current behavior space without rewriting historical evidence.Troubleshooting / Recovery LibraryUnbalanced CoverageFix benchmarks that overrepresent easy or common cases while missing important behavior.Troubleshooting / Recovery LibrarySynthetic Cases That Feel UnrealisticImprove generated candidates whose surface details or behavior assumptions do not represent plausible target-system use.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Learning Manual The Teammately correctness lifecycle Follow specialist AI work from project foundations through coverage, elicitation, construction, evaluation, and improvement. Coverage Engineering Coverage Gaps Find missing or underrepresented behavior areas before benchmark evidence becomes misleading. Governance and Reproducibility Versioning and Staleness Know when correctness objects changed and when old evidence may need review.
AI context