# Coverage Stories Generated: 2026-09-13T04:39:10.648Z Source build: local Canonical docs: https://teammately.ai/docs --- id: coverage-management.coverage-stories title: Coverage Stories summary: Organize benchmark coverage intent into governed Stories and testable facet tuples. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/coverage-stories --- # Coverage Stories ## Prerequisites - Ready coverage setup or a clearly justified manual Story. - Coverage Facets that can name the intended behavior slice. A Coverage Story turns benchmark intent into a coherent behavior slice. It explains what matters, why it matters, and which facet combinations must be exercised without pretending that a chart category alone describes a real product situation. Stories can be created manually or proposed by generation. Their lifecycle is `draft`, `active`, or `archived`, and their origin remains visible as AI-generated or manual. A Story contains a title, description, rationale, intent, budget, referenced facets, suggested Topic Groups, and one or more tuples. ## Make tuples testable Each tuple names a smaller test obligation through its title, `must_prove` statement, facet references, target, accepted count, status, and flags. Write `must_prove` so Case Review can decide whether a candidate actually supplies the required evidence. Avoid vague goals such as “good edge cases.” Name the actor, conflict, evidence, constraint, or transformation that distinguishes the tuple. Targets express desired evidence volume; accepted counts report materialized evidence. Neither number proves quality. A tuple can meet its count while still containing redundant or unrealistic Cases, so review remains mandatory. ## Edit and govern the Story map Use the editor to create or revise a Story and add, edit, or remove tuples. Archive a Story whose behavior is no longer in benchmark scope. Do not delete or rewrite the rationale merely because the current dataset already covers it; that rationale explains why the evidence exists. When generation is running, partial, failed, or based on changed setup, read the displayed generation state before acting. A generated Story remains a proposal until the saved story and tuple structure reflects the benchmark intent. {% example-demo title="Example: superseded-source conflict" %} A Story covers answers that cite a plausible but superseded policy. One tuple must prove that the candidate detects the date conflict; another must prove that it asks for clarification when the current source is incomplete. Their separate targets prevent several near-duplicate date cases from masquerading as coverage of both behaviors. {% /example-demo %} ## Object and state changes Creating or editing changes the saved Story and tuple map. Archiving removes a Story from active planning while preserving it. Generation can propose Stories but does not accept Cases into the dataset. ## Success criteria - Every active Story has a clear rationale and testable tuples. - Targets and accepted counts remain distinguishable. - Story origin and generation freshness are visible. ## Common failure modes - Writing tuples that cannot be judged in Case Review. - Treating target count as evidence quality. - Merging distinct Topic or facet obligations into vague coverage prose. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Management](/docs/coverage-management) - [Project Topics](/docs/coverage-engineering/project-topics) - [Case Construction Patterns](/docs/coverage-engineering/case-construction-patterns) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Stale Dimensions](/docs/troubleshooting/stale-dimensions) {% /related-card-grid %} ## Source confidence Code-backed: the current Coverage Stories page, editor, and types define Story origin, lifecycle, fields, tuple structure, counts, and generation presentation. --- id: coverage-management.overview title: Coverage Management summary: Manage benchmark coverage from setup through representation, Coverage Stories, case review, Case Foundry, and contribution requests. kind: concept product_area: coverage_management status: stable updated: 2026-09-07 canonical: /docs/coverage-management --- # Coverage Management Coverage Management is the benchmark-scoped workspace for deciding whether the current dataset represents the behavior space the benchmark is meant to test. It connects project Coverage Facets to dataset representation, coverage guidance, Coverage Stories, case preparation, and expert contribution requests. ## Definition Project Coverage Facets define reusable Dimensions, Project Topics, and Case Construction Patterns. Coverage Management applies those foundations to one benchmark. **Get Started** establishes benchmark coverage guidance. The overview shows representation and operational status. Coverage Stories organize meaningful slices or gaps. Case Review inspects prepared cases and materials. Case Foundry coordinates case construction work. The goal is to make missing or thin behavior explicit before evaluation evidence is trusted. Coverage Management does not replace Benchmark Datasets; it explains and improves the representation of the selected data. ## Decision checkpoint | Observation | Use | Next durable result | | --- | --- | --- | | Benchmark purpose or guidance is missing | Get Started | Saved coverage setup and readiness | | A facet tuple is thin or absent | Representation and Coverage Stories | Named coverage need and intended evidence | | More cases are needed | Case Foundry | Bounded construction work tied to the gap | | Generated cases may be unclear | Case Review | Reviewed case and material quality | | Specialist judgment is required | Contribution request from coverage context | Benchmark-scoped Expert Contribution | | Coverage changed materially | Benchmark Datasets | Updated selection and snapshot boundary | ## Coverage Stories and case work A Coverage Story gives a gap or behavior slice an operational narrative: why it matters, which facet combinations define it, what evidence exists, and what sourcing work remains. It should be concrete enough to guide case construction and expert attention. Case Foundry can prepare case work from that structure. Case Review checks the resulting inputs and generated materials before they enter trusted dataset evidence. AI assistance can accelerate preparation, but selection and benchmark interpretation remain explicit human and product-state decisions. ## Coverage and correctness Coverage gaps sometimes reveal missing correctness rather than missing cases. If experts cannot say how a represented situation should be judged, request an Expert Contribution and update policies or rubrics. If the standard is clear but no case exercises it, use Weave and Case Foundry. If cases exist but are not selected or snapshotted, use Benchmark Datasets. This routing prevents Comparison Directions, Contribution-scoped agent behavior, and coverage structure from being mixed together. Comparison Directions guide response variation; Coverage Facets and Coverage Stories describe the benchmark behavior space. {% example-demo title="Conflicting-source story" %} Representation shows that the benchmark covers current-source questions but almost never combines them with a plausible superseded document. A Coverage Story names the conflict pattern, relevant source-freshness and impact facets, and the desired case count. Case Foundry prepares candidates, Case Review rejects unrealistic material, and the accepted cases enter a new dataset snapshot. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Set up benchmark coverage](/docs/coverage-management/get-started) - [Work with Coverage Stories](/docs/coverage-management/coverage-stories) - [Run Case Foundry](/docs/coverage-management/case-foundry) - [Review prepared Cases](/docs/coverage-management/case-review) - [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage) - [Work with Benchmark Datasets](/docs/benchmark-datasets) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Engineering](/docs/coverage-engineering) - [Cases](/docs/assets/cases) - [Comparison Directions](/docs/assets/comparison-directions) {% /related-card-grid %} ## Source confidence Code-backed: the active benchmark coverage routes expose setup, overview, Coverage Stories, Case Review, Case Foundry integration, realtime state, and contribution-request entry points. --- id: coverage-management.get-started title: Set Up Benchmark Coverage summary: Define benchmark intent, facet handling, artifact preferences, and evidence requirements before generating coverage work. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/get-started --- # Set Up Benchmark Coverage ## Prerequisites - Project Coverage Facets and Input Schema are available. - The benchmark intent and evidence risk can be stated concretely. **Get Started** records the coverage contract that drives Coverage Stories and Case Foundry. Complete it before treating generated coverage work as aligned to the benchmark. ## Define the intent and facet treatment Describe the benchmark intent and concrete requirements. For each Dimension ontology value, choose the benchmark role required by the setup. Configure Topic Group handling and Case Construction Pattern behavior rather than assuming every active project facet must be represented equally. Dimension roles, Topic Group handling, and Pattern modes are different controls. A required Dimension value constrains represented behavior. A Topic Group can require every Topic, require group-level coverage, provide guidance, or be excluded. A Pattern can be left to the system, preferred, or avoided. Preserve those distinctions when explaining the resulting coverage plan. ## Define artifact and evidence expectations For image, document, tabular, presentation, source-text, and audio artifacts, choose **mandatory**, **optional**, or **never**. Set portfolio limits and accepted formats so construction does not create unsupported or gratuitous material. The evidence profile can specify actor, workflow, grounding, evidence carriers, difficulty, transformation, and data-handling expectations. These fields make a behavior testable. They are not decorative prose: Case Review uses them to judge whether a prepared candidate proves the intended situation. ## Save and check readiness Setup moves through `draft`, `ready`, `generated`, `changed_since_generation`, and `archived` states. Resolve the readiness guidance before generation. If the setup changes after stories or cases were generated, treat the previous work as based on an older input rather than silently presenting it as current. A coverage guideline can apply to `foundry_only` or `overall_coverage`. Overall coverage can require provenance backfill for existing Cases. The coverage compiler can preview reconciled revisions, but an operator confirms the durable update. > Generation boundary > > Saving setup does not create trusted Cases or a Dataset Snapshot. It defines the instructions and evidence profile for downstream story and case work. ## Object and state changes Saving creates or revises benchmark-scoped coverage setup and its readiness status. Generation records which setup revision it used. Archiving stops the setup from acting as the current contract without erasing history. ## Success criteria - Intent, requirements, facet treatment, artifacts, and evidence profile agree. - Readiness is explicit and downstream generation can identify the exact setup. - Overall-coverage provenance needs are handled deliberately. ## Common failure modes - Requiring every active facet without regard to benchmark intent. - Marking unsupported artifacts mandatory. - Editing setup after generation and ignoring the stale result. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Management](/docs/coverage-management) - [Project Input Schema](/docs/project-settings/input-schema) - [Coverage Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Unrealistic synthetic Cases](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} ## Source confidence Code-backed: the active setup page, types, readiness rules, and save states define the benchmark-scoped coverage contract and its lifecycle. --- id: coverage-management.case-foundry title: Case Foundry summary: Generate or update bounded case candidates from the saved coverage setup and Story map. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/case-foundry --- # Case Foundry ## Prerequisites - Ready coverage setup, active Stories, and actionable tuple targets. - A current input snapshot whose freshness can be verified. Case Foundry coordinates bounded case preparation from the saved coverage setup and Coverage Stories. Use **Generate** for the first run and **Update** after the governed inputs change. A blocked action means required setup, readiness, or upstream evidence is not yet available. ## Run states and freshness A Foundry run reports `QUEUED`, `RUNNING`, `COMPLETED`, `PARTIAL`, `FAILED`, or `CANCELLED`. Keep the run identity and input snapshot together when diagnosing it. Completion can report Cases added, synthesized, or retrieved; these counts explain construction activity, not acceptance into trusted benchmark evidence. The workspace distinguishes a fresh result from one whose input snapshot changed. If coverage setup, Story structure, or relevant dataset context changed after the run began, update the Foundry work. Do not use an old completion badge as proof that the current coverage contract has been handled. ## Before and after Foundry Before running, make the setup ready, activate the intended Stories, and check tuple targets. If a specialist must settle an ambiguous standard, request an Expert Contribution instead of generating around the ambiguity. After running, open Case Review. Prepared candidates start as reviewable material rather than automatically becoming durable benchmark Cases. Inspect their evidence fit, sources, facet traces, inputs, and generated artifacts. Remove weak or redundant candidates and resolve material failures before inclusion. > Case Foundry is not a Dataset Snapshot > > Foundry prepares candidates. Case Review admits suitable Cases into the current benchmark set, and Benchmark Datasets creates the immutable Snapshot used for evaluation. ## Failure handling For a partial or failed run, preserve successful bounded work, read the failure detail, and retry only the missing scope when the product offers that action. Cancellation stops the current operation; it does not roll back Cases already materialized by a completed portion. Recheck freshness after any retry. ## Object and state changes A Foundry action creates a run and can prepare, synthesize, retrieve, or add candidate Cases. Update creates new bounded work from changed input. The run does not create a Dataset Snapshot or bypass Case Review. ## Success criteria - Terminal status and input freshness are known. - Summary counts are interpreted as construction activity. - Prepared candidates move to review rather than automatic trust. ## Common failure modes - Treating `COMPLETED` as Case acceptance. - Retrying stale work without updating its input. - Generating around an unresolved correctness question. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Management](/docs/coverage-management) - [Coverage Stories](/docs/coverage-management/coverage-stories) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unrealistic synthetic Cases](/docs/troubleshooting/unrealistic-synthetic-cases) - [Unclear Cases](/docs/troubleshooting/unclear-cases) {% /related-card-grid %} ## Source confidence Code-backed: the active Foundry API, run types, and workspace hook define actions, states, freshness, and summary counts. Case acceptance is verified in the separate Case Review surface. --- id: coverage-management.case-review title: Case Review summary: Prepare, inspect, refine, and admit Case candidates and generated materials into the benchmark dataset. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/case-review --- # Case Review ## Prerequisites - Prepared candidates tied to a frozen setup, Story map, input contract, and output formats. - Permission to admit or remove candidates from current benchmark membership. Case Review is the admission boundary between prepared coverage candidates and the current benchmark dataset. Preparation freezes the relevant setup, Stories, input contract, and output formats so a candidate can be judged against the instructions that produced it. Preparation can move through planning, searching, evaluating, generating, synthesizing, materializing, ready, partial, failed, and cancelled states. Read this state together with the frozen input rather than assuming every visible card is complete. ## Review candidates Candidates begin included for review. Remove a candidate when it is weak, redundant, misplaced, unsupported, or does not prove its tuple. Inspect its fit explanation, source and facet trace, input content, materials, and relationship to the tuple target. Inclusion should mean the candidate is suitable to enter the current benchmark set, not merely that generation succeeded. For one tuple, **Generate more** appends candidates using count and operator instructions. **Regenerate** can reuse or synthesize source evidence, or run in synthesize-only mode. These actions have different provenance implications; preserve the displayed source relationship when deciding which candidate to keep. ## Resolve generated materials Artifact expectations from coverage setup can be mandatory or optional. Material resolution reports generating, verifying, ready, failed, or skipped. A mandatory artifact failure blocks trustworthy inclusion. An optional artifact may be skipped when the Case remains coherent without it. Retry uses the original material specification. If the specification itself is wrong, correct the governed setup or candidate design rather than repeatedly retrying the same request. Verify that a ready file actually supports the Case and conforms to Project Input Schema. ## Complete the review Accepted included candidates materialize into the benchmark's current Case set. Then inspect Dataset Representation and create a new Snapshot only after evaluator readiness and Snapshot blockers are clear. Existing Snapshots remain unchanged. {% example-demo title="Example: rejecting decorative evidence" %} A tuple requires the candidate system to reconcile two contradictory tables. One prepared Case has a table that never affects the answer, while another requires comparing two columns and citing the newer record. The reviewer removes the decorative Case, verifies the second table, and admits only the candidate that proves the intended transformation. {% /example-demo %} ## Object and state changes Inclusion and removal change the review selection; accepted included candidates materialize into the current Case set. Generate-more and regenerate create new candidates. Material retries update resolution state without changing the original specification. ## Success criteria - Included Cases prove their tuples and have usable provenance. - Mandatory materials are ready and verified. - Dataset membership and the next Snapshot reflect only accepted work. ## Common failure modes - Keeping decorative or redundant candidates to meet a count. - Treating material-generation success as Case quality. - Assuming acceptance changed an existing Snapshot. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Project Input Schema](/docs/project-settings/input-schema) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) - [Dataset Snapshots](/docs/benchmark-datasets/snapshots) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unclear Cases](/docs/troubleshooting/unclear-cases) - [Unrealistic synthetic Cases](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} ## Source confidence Code-backed: the active Case Review page, preparation modal, tuple actions, and Generated Materials component define frozen inputs, run states, inclusion, generation modes, and artifact resolution.