# Coverage Stories Generated: 2026-11-03T08:09:10.823Z Source build: local Canonical docs: https://teammately.ai/docs --- id: coverage-management.coverage-stories title: Coverage Stories summary: Describe meaningful input situations and testable facet combinations used in Coverage Design and Case preparation. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/coverage-stories --- # Coverage Stories Stories explain the situation behind a coverage requirement. They connect Benchmark intent to testable facet combinations and the Cases prepared to exercise them. ## Prerequisites - An identified Benchmark intent and current coverage requirements. - Dimensions, Topics or Scenarios that describe the relevant distinctions. - A Case input contract that can represent the situation. ## Make a Story concrete Describe the actor, request, source condition, conflict or constraint that changes expected behavior. A Story should help an operator explain why a slice matters without treating a chart category as a complete situation. Tuples name smaller obligations. Their statement of what must be demonstrated, facet references and intended count guide preparation and review. A tuple's requested count does not prove that its Cases are realistic or distinct. ## Read Stories with Canvas Use Canvas to inspect current requirements and selected Cases. Stories supply planning context; requirement assessments supply current matched, unknown and remaining counts. Do not substitute a Story budget for actual dataset membership. Where a Story is created or edited through the available controls, preserve its rationale and relationship to the relevant facets. An archived Story is history rather than a new instruction to remove every related Case. ## Materials Project Input Schema supplies allowed artifacts and required material keys. Story planning can request an appropriate allowed artifact. It cannot disable required input, add an unaccepted family or create another materials policy. ## Object and state changes Story edits change current planning context. Prepared work records the Story and input contract it accepted. Existing Snapshots and completed Runs retain their original Cases and criteria. ## Success criteria - A reviewer can tell what situation each tuple must demonstrate. - Facet references identify the intended project concepts. - Counts are interpreted alongside actual Case quality and membership. - Current requirements and captured preparation inputs are distinguished. ## Common failure modes - Using a generic title as the entire requirement. - Counting near-duplicate Cases as distinct coverage. - Reconstructing an accepted preparation from today's edited Story. - Treating Story generation as automatic Case adoption. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Coverage Design Settings](/docs/coverage-design/settings) - [Case Foundry](/docs/coverage-design/case-foundry) - [Review Prepared Cases](/docs/coverage-design/case-review) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: coverage-management.overview title: Coverage Design summary: Use Canvas to define required input coverage and Inbox to decide how new findings affect benchmark membership. kind: concept product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design --- # Coverage Design Coverage Design applies reusable project facets to one Benchmark. Use **Canvas** to plan and inspect input coverage, and **Inbox** to decide whether new findings should change the selected Cases. ## Definition A coverage requirement names an input situation and how it should be represented. It can require a minimum Case count or record an observation. Dimension ontology values, Topics and Scenarios identify the intended slice; distribution settings describe its share of the active benchmark population. Canvas connects those requirements to actual selected Cases. A large total does not establish that an important combination is present, and unknown classification does not establish absence. ## Decision checkpoint | Need | Use | Result | | --- | --- | --- | | Define the intended behavior space | Canvas requirements and settings | Explicit coverage obligations | | Inspect a slice or combination | Canvas selection and matrix | Matching Cases, unknowns and remaining need | | Prepare more Cases | Case preparation | Reviewable candidates and materials | | Act on an expert or Fieldloop finding | Inbox | An explicit adoption decision | | Freeze the selected population | Benchmark Datasets → Snapshots | An immutable Benchmark Version | ## Work from Canvas Describe the Benchmark intent, select the relevant facets and inspect count or distribution requirements. Examine Cases behind a count before deciding that a gap is resolved. Distinguish requirements from observations and current membership from available project Cases. Stories and their tuples describe meaningful situations used in planning and Case preparation. Case Foundry coordinates preparation; Case Review is where prepared inputs and materials are inspected. These mechanisms support Canvas work rather than replacing its current requirement and membership view. ## Materials and input shape Project Input Schema owns allowed artifact families, accepted formats and required material keys. Coverage Design does not add another Materials configuration. A tuple can require an allowed artifact but cannot remove a required input or enable an unsupported format. ## Act from Inbox Inbox links attributable input-coverage findings to possible dataset changes. Review each observation or proposed Case separately. Generating a Case stages another item; it does not automatically add membership. A resolved report can contain declined or withdrawn items. Existing Snapshots and Runs retain their recorded population. Create a new Snapshot when changed membership or standards should define a later evaluation. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design Settings](/docs/coverage-design/settings) - [Input Coverage Inbox](/docs/coverage-design/inbox) - [Coverage Stories](/docs/coverage-design/coverage-stories) - [Review Prepared Cases](/docs/coverage-design/case-review) - [Benchmark Datasets](/docs/benchmark-datasets) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} --- id: coverage-management.get-started title: Coverage Design Settings summary: Set benchmark intent, facet handling, and coverage requirements while inheriting materials from Project Input Schema. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/settings --- # Coverage Design Settings Use the settings and requirement controls in **Coverage Design → Canvas** to state which input situations the Benchmark must represent. ## Prerequisites - A selected Benchmark and permission to manage it. - Project facets that describe the intended distinctions. - A saved Project Input Schema. - A concrete statement of Benchmark intent. ## Define intent and facet handling Explain the intended behavior and the boundaries that matter. Choose Dimension ontology roles, Topic Group handling and Scenario steering for their different purposes. A required ontology value identifies a coverage obligation. A Topic Group can require every member Topic, require group-level representation, provide guidance or be excluded. Scenario steering controls which reusable constructions are preferred, avoided or left to the system. These controls do not make every active project facet mandatory. ## Define requirements Name each situation and its facets. Use a minimum Case count when a specific volume is required. Distribution bounds or targets express the desired share of the active benchmark population. Read matched, unknown and remaining counts separately. An unknown classification is not a negative match. Inspect the actual selected Cases before concluding that a requirement is met. ## Inherit materials Allowed artifacts, file formats and required Case materials come from **Project Input Schema**. Change them there when the project contract needs to change. Canvas has no separate Materials settings. A required image material remains required even if an old Benchmark setting had excluded images. Accepting PNG and JPEG permits alternatives; it does not require both. New preparation captures the effective schema-derived contract. Accepted earlier work retains its captured settings. ## Object and state changes Saving changes current benchmark coverage intent or requirements. It does not approve a generated Case, change an existing Snapshot or rewrite accepted preparation inputs. Actual Case adoption changes editable membership through its owning action. ## Success criteria - Requirements describe meaningful situations and the intended population. - Count and distribution expectations are explicit. - Unknown classifications remain visible. - Materials agree with the project contract. ## Common failure modes - Treating a Topic Group as though it merges its Topics. - Requiring every active facet without a reason. - Attempting to override required materials at Benchmark scope. - Declaring a gap closed from a total count without inspecting the Cases. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Dimensions and Ontology](/docs/coverage-engineering/dimensions-ontology) - [Scenarios](/docs/coverage-engineering/scenarios) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: coverage-management.case-foundry title: Case Foundry summary: Prepare Case candidates for Coverage Design from current requirements and captured project input settings. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/case-foundry --- # Case Foundry Case Foundry coordinates Case preparation for the situations defined in Coverage Design. Read preparation progress separately from reviewed Case quality and benchmark membership. ## Prerequisites - Coverage intent and requirements that identify the missing situation. - Relevant Story or tuple planning context. - Project Input Schema and the materials needed by that contract. - Permission to prepare and review Cases. ## Prepare candidates Start from Canvas and the intended coverage slice. Confirm whether the work should reuse source material, synthesize a situation or refine existing candidates through the available preparation controls. Preparation captures the relevant planning and input settings. Its queued, running, partial, completed, failed or cancelled state describes that accepted operation. A completed operation is not a statement that every current requirement is now satisfied. Project Input Schema owns allowed artifact families, formats and required material keys. An unavailable required family produces an actionable preparation error. Do not replace required material with text or change the project contract merely to make generation appear complete. ## Review the result Open Case Review to inspect input content, generated materials, provenance and the situation demonstrated by each candidate. Read added, synthesized or retrieved counts as preparation activity. Confirm actual selected membership in Canvas or Benchmark Datasets. If requirements or source material changed after preparation started, inspect the captured inputs before requesting another operation. Historical completion does not establish freshness against today's settings. ## Object and state changes Preparation creates an attributable operation and reviewable candidate material. Review and explicit membership operations determine which Cases enter the editable benchmark. Existing Snapshots remain unchanged. ## Success criteria - Candidate preparation is tied to a named input need. - The captured contract and actual required materials agree. - Partial or failed work remains distinguishable from successful output. - Reviewed membership, rather than generation volume, explains progress. ## Common failure modes - Generating around an unresolved correctness question. - Treating operation completion as Case approval. - Using obsolete Benchmark material preferences instead of Project Input Schema. - Assuming an old preparation handled newly edited requirements. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Review Prepared Cases](/docs/coverage-design/case-review) - [Project Input Schema](/docs/project-settings/input-schema) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) - [Unclear Cases](/docs/troubleshooting/unclear-cases) {% /related-card-grid %} --- id: coverage-management.case-review title: Review Prepared Cases summary: Inspect generated inputs and materials against their captured coverage purpose before relying on benchmark membership. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/case-review --- # Review Prepared Cases Review prepared Cases for the input situation they actually demonstrate. Preparation success, current inclusion and sufficient coverage are separate conclusions. ## Prerequisites - Prepared candidates and their captured coverage context. - Permission to manage benchmark membership. - The Project Input Schema and required materials used for preparation. ## Inspect candidates Read the canonical input, source relationship, facet assignments and stated fit to the tuple or requirement. Check required files and values rather than assuming a completed generation produced usable materials. Remove a candidate that is redundant, unsupported or misplaced through the available controls. Inclusion should reflect a deliberate decision about the current editable dataset. It does not change an older Snapshot. For a tuple that still lacks suitable Cases, generation can append candidates and regeneration can replace the preparation proposal according to its displayed source mode. Preserve whether material was retrieved, reused or synthesized. ## Materials and compatibility Project Input Schema owns material keys, accepted families and extensions. A required material must be present and usable. Accepted extensions are alternatives, not a demand to generate every listed format. Generated-material status and Case status may settle at different times. A partial or failed attachment is not a complete Case. Use the reported failure to correct the owning source or contract rather than silently dropping the required material. ## Object and state changes Review changes candidate inclusion or editable membership through the owning operation. Additional preparation records another captured request. A Snapshot freezes the selected Cases only when explicitly created. Cases staged by an Inbox generation have a separate adoption decision in Inbox. Completing generation alone does not add those Cases or resolve the source observation. ## Success criteria - The input demonstrates its intended situation without invented assumptions. - Required materials are inspectable and conform to the captured contract. - Current membership matches the review decision. - Repeated or generated Cases retain their source relationship. ## Common failure modes - Accepting a candidate because its card exists. - Using count completion as proof of distinct coverage. - Confusing generated material with approved correctness. - Treating a Case correction as a mutation of historical Runs. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Input Coverage Inbox](/docs/coverage-design/inbox) - [Case Foundry](/docs/coverage-design/case-foundry) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unclear Cases](/docs/troubleshooting/unclear-cases) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: troubleshooting.unbalanced-coverage title: Unbalanced Coverage summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior. kind: error product_area: troubleshooting status: stable updated: 2026-10-30 canonical: /docs/troubleshooting/unbalanced-coverage --- # Unbalanced Coverage Use this when benchmark results overrepresents some behavior and misses other behavior that matters before human review. ## Symptom A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent. ## Likely causes - Case import mirrored production volume instead of product risk. - Dimensions or metadata do not represent the behavior areas that matter. - Synthetic cases were added in one area but not reviewed across the full risk boundary. - Coverage was not refreshed after new signals, policy changes, or failures. ## Diagnostic checks - Inspect **Benchmark Datasets → Representation** and the Coverage Design overview by the configured Coverage Facets. - Look for policies or rubrics with very few applicable cases. - Compare production or expert signals with benchmark case distribution. - Inspect whether new cases are candidate-only or actually in use. ## Fix - Add representative, boundary, or adversarial cases for undercovered areas. - Refresh dimensions or metadata before judging balance. - Review and promote candidate cases into the benchmark only after context is complete. - Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance. ## Prevention - Define coverage targets before benchmark runs. - Track candidate and in-use cases separately. - Review coverage after policy, rubric, or product behavior changes. - Do not use aggregate scores without a coverage note. ## Related task pages {% related-card-grid title="Related task pages" %} - [Dataset Representation](/docs/benchmark-datasets/representation) - [Coverage Design Settings](/docs/coverage-design/settings) - [Coverage Stories](/docs/coverage-design/coverage-stories) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Case Pool surface](/docs/object-model/case-pool) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %} --- id: troubleshooting.unrealistic-synthetic-cases title: Synthetic Cases That Feel Unrealistic summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use. kind: error product_area: troubleshooting status: stable updated: 2026-10-30 canonical: /docs/troubleshooting/unrealistic-synthetic-cases --- # Synthetic Cases That Feel Unrealistic ## Symptoms - Generated language sounds artificial or describes a user who would not exist in the product. - Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere. - The Case combines incompatible ontology values or violates product constraints. - Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark. - Many generated candidates repeat the same template with superficial wording changes. ## Likely causes - The synthesis request lacks representative source Cases or product constraints. - A coverage tuple combines values that cannot coexist in the target system. - A Comparison Direction is too broad, stale, or duplicative. - Generation is being judged as finished benchmark results instead of candidate material. ## Diagnose the generation boundary 1. Identify the entry surface. **Assets → Cases → Synthesize** creates reusable project Case candidates; **Coverage Design → Case Foundry** coordinates candidates for a named benchmark need. 2. Compare the generated Case with real in-use Cases from the same behavior region. 3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires different inputs or directions. 4. Check whether the generator lacked constraints that operators hold only in private memory. 5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not selected Benchmark Dataset supporting information. ## Fix - Add representative source Cases or explicit product constraints to the synthesis context. - Narrow, edit, archive, or delete a Comparison Direction that produces implausible combinations. - Generate from a specific missing coverage tuple rather than asking for generic diversity. - Edit a promising candidate when its behavioral boundary is sound and only presentation is weak. - Reject candidates whose premise is impossible or whose source context cannot be justified. - Review duplicates before adding any generated Case to a Benchmark. Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use. ## Prevention Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Keep Comparison Directions narrow, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot. {% example-demo title="Example: impossible refund request" %} Synthesis produces a Case where an anonymous visitor requests an enterprise-contract refund and supplies internal approval metadata in chat. The tuple is valuable—exception request with conflicting authority—but the realization is impossible. The team adds authentication and channel constraints, narrows the Comparison Direction, and regenerates. The realistic candidate still goes through Case review before selection. {% /example-demo %} ## Related task pages {% related-card-grid title="Related workflows" %} - [Synthesize Cases](/docs/coverage-engineering/synthesize-cases) - [Comparison Directions](/docs/assets/comparison-directions) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Case Pool](/docs/coverage-engineering/case-pool) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Case Pool surface](/docs/object-model/case-pool) - [Cases](/docs/object-model/cases) - [Coverage dimensions](/docs/object-model/coverage-dimensions) {% /related-card-grid %}