# Case Foundry Generated: 2026-09-13T04:36:26.434Z Source build: local Canonical docs: https://teammately.ai/docs --- id: coverage-management.case-foundry title: Case Foundry summary: Generate or update bounded case candidates from the saved coverage setup and Story map. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/case-foundry --- # Case Foundry ## Prerequisites - Ready coverage setup, active Stories, and actionable tuple targets. - A current input snapshot whose freshness can be verified. Case Foundry coordinates bounded case preparation from the saved coverage setup and Coverage Stories. Use **Generate** for the first run and **Update** after the governed inputs change. A blocked action means required setup, readiness, or upstream evidence is not yet available. ## Run states and freshness A Foundry run reports `QUEUED`, `RUNNING`, `COMPLETED`, `PARTIAL`, `FAILED`, or `CANCELLED`. Keep the run identity and input snapshot together when diagnosing it. Completion can report Cases added, synthesized, or retrieved; these counts explain construction activity, not acceptance into trusted benchmark evidence. The workspace distinguishes a fresh result from one whose input snapshot changed. If coverage setup, Story structure, or relevant dataset context changed after the run began, update the Foundry work. Do not use an old completion badge as proof that the current coverage contract has been handled. ## Before and after Foundry Before running, make the setup ready, activate the intended Stories, and check tuple targets. If a specialist must settle an ambiguous standard, request an Expert Contribution instead of generating around the ambiguity. After running, open Case Review. Prepared candidates start as reviewable material rather than automatically becoming durable benchmark Cases. Inspect their evidence fit, sources, facet traces, inputs, and generated artifacts. Remove weak or redundant candidates and resolve material failures before inclusion. > Case Foundry is not a Dataset Snapshot > > Foundry prepares candidates. Case Review admits suitable Cases into the current benchmark set, and Benchmark Datasets creates the immutable Snapshot used for evaluation. ## Failure handling For a partial or failed run, preserve successful bounded work, read the failure detail, and retry only the missing scope when the product offers that action. Cancellation stops the current operation; it does not roll back Cases already materialized by a completed portion. Recheck freshness after any retry. ## Object and state changes A Foundry action creates a run and can prepare, synthesize, retrieve, or add candidate Cases. Update creates new bounded work from changed input. The run does not create a Dataset Snapshot or bypass Case Review. ## Success criteria - Terminal status and input freshness are known. - Summary counts are interpreted as construction activity. - Prepared candidates move to review rather than automatic trust. ## Common failure modes - Treating `COMPLETED` as Case acceptance. - Retrying stale work without updating its input. - Generating around an unresolved correctness question. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Management](/docs/coverage-management) - [Coverage Stories](/docs/coverage-management/coverage-stories) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unrealistic synthetic Cases](/docs/troubleshooting/unrealistic-synthetic-cases) - [Unclear Cases](/docs/troubleshooting/unclear-cases) {% /related-card-grid %} ## Source confidence Code-backed: the active Foundry API, run types, and workspace hook define actions, states, freshness, and summary counts. Case acceptance is verified in the separate Case Review surface. --- id: coverage-management.coverage-stories title: Coverage Stories summary: Organize benchmark coverage intent into governed Stories and testable facet tuples. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/coverage-stories --- # Coverage Stories ## Prerequisites - Ready coverage setup or a clearly justified manual Story. - Coverage Facets that can name the intended behavior slice. A Coverage Story turns benchmark intent into a coherent behavior slice. It explains what matters, why it matters, and which facet combinations must be exercised without pretending that a chart category alone describes a real product situation. Stories can be created manually or proposed by generation. Their lifecycle is `draft`, `active`, or `archived`, and their origin remains visible as AI-generated or manual. A Story contains a title, description, rationale, intent, budget, referenced facets, suggested Topic Groups, and one or more tuples. ## Make tuples testable Each tuple names a smaller test obligation through its title, `must_prove` statement, facet references, target, accepted count, status, and flags. Write `must_prove` so Case Review can decide whether a candidate actually supplies the required evidence. Avoid vague goals such as “good edge cases.” Name the actor, conflict, evidence, constraint, or transformation that distinguishes the tuple. Targets express desired evidence volume; accepted counts report materialized evidence. Neither number proves quality. A tuple can meet its count while still containing redundant or unrealistic Cases, so review remains mandatory. ## Edit and govern the Story map Use the editor to create or revise a Story and add, edit, or remove tuples. Archive a Story whose behavior is no longer in benchmark scope. Do not delete or rewrite the rationale merely because the current dataset already covers it; that rationale explains why the evidence exists. When generation is running, partial, failed, or based on changed setup, read the displayed generation state before acting. A generated Story remains a proposal until the saved story and tuple structure reflects the benchmark intent. {% example-demo title="Example: superseded-source conflict" %} A Story covers answers that cite a plausible but superseded policy. One tuple must prove that the candidate detects the date conflict; another must prove that it asks for clarification when the current source is incomplete. Their separate targets prevent several near-duplicate date cases from masquerading as coverage of both behaviors. {% /example-demo %} ## Object and state changes Creating or editing changes the saved Story and tuple map. Archiving removes a Story from active planning while preserving it. Generation can propose Stories but does not accept Cases into the dataset. ## Success criteria - Every active Story has a clear rationale and testable tuples. - Targets and accepted counts remain distinguishable. - Story origin and generation freshness are visible. ## Common failure modes - Writing tuples that cannot be judged in Case Review. - Treating target count as evidence quality. - Merging distinct Topic or facet obligations into vague coverage prose. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Management](/docs/coverage-management) - [Project Topics](/docs/coverage-engineering/project-topics) - [Case Construction Patterns](/docs/coverage-engineering/case-construction-patterns) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Stale Dimensions](/docs/troubleshooting/stale-dimensions) {% /related-card-grid %} ## Source confidence Code-backed: the current Coverage Stories page, editor, and types define Story origin, lifecycle, fields, tuple structure, counts, and generation presentation. --- id: coverage-management.case-review title: Case Review summary: Prepare, inspect, refine, and admit Case candidates and generated materials into the benchmark dataset. kind: task product_area: coverage_management status: stable updated: 2026-08-22 canonical: /docs/coverage-management/case-review --- # Case Review ## Prerequisites - Prepared candidates tied to a frozen setup, Story map, input contract, and output formats. - Permission to admit or remove candidates from current benchmark membership. Case Review is the admission boundary between prepared coverage candidates and the current benchmark dataset. Preparation freezes the relevant setup, Stories, input contract, and output formats so a candidate can be judged against the instructions that produced it. Preparation can move through planning, searching, evaluating, generating, synthesizing, materializing, ready, partial, failed, and cancelled states. Read this state together with the frozen input rather than assuming every visible card is complete. ## Review candidates Candidates begin included for review. Remove a candidate when it is weak, redundant, misplaced, unsupported, or does not prove its tuple. Inspect its fit explanation, source and facet trace, input content, materials, and relationship to the tuple target. Inclusion should mean the candidate is suitable to enter the current benchmark set, not merely that generation succeeded. For one tuple, **Generate more** appends candidates using count and operator instructions. **Regenerate** can reuse or synthesize source evidence, or run in synthesize-only mode. These actions have different provenance implications; preserve the displayed source relationship when deciding which candidate to keep. ## Resolve generated materials Artifact expectations from coverage setup can be mandatory or optional. Material resolution reports generating, verifying, ready, failed, or skipped. A mandatory artifact failure blocks trustworthy inclusion. An optional artifact may be skipped when the Case remains coherent without it. Retry uses the original material specification. If the specification itself is wrong, correct the governed setup or candidate design rather than repeatedly retrying the same request. Verify that a ready file actually supports the Case and conforms to Project Input Schema. ## Complete the review Accepted included candidates materialize into the benchmark's current Case set. Then inspect Dataset Representation and create a new Snapshot only after evaluator readiness and Snapshot blockers are clear. Existing Snapshots remain unchanged. {% example-demo title="Example: rejecting decorative evidence" %} A tuple requires the candidate system to reconcile two contradictory tables. One prepared Case has a table that never affects the answer, while another requires comparing two columns and citing the newer record. The reviewer removes the decorative Case, verifies the second table, and admits only the candidate that proves the intended transformation. {% /example-demo %} ## Object and state changes Inclusion and removal change the review selection; accepted included candidates materialize into the current Case set. Generate-more and regenerate create new candidates. Material retries update resolution state without changing the original specification. ## Success criteria - Included Cases prove their tuples and have usable provenance. - Mandatory materials are ready and verified. - Dataset membership and the next Snapshot reflect only accepted work. ## Common failure modes - Keeping decorative or redundant candidates to meet a count. - Treating material-generation success as Case quality. - Assuming acceptance changed an existing Snapshot. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Project Input Schema](/docs/project-settings/input-schema) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) - [Dataset Snapshots](/docs/benchmark-datasets/snapshots) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unclear Cases](/docs/troubleshooting/unclear-cases) - [Unrealistic synthetic Cases](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} ## Source confidence Code-backed: the active Case Review page, preparation modal, tuple actions, and Generated Materials component define frozen inputs, run states, inclusion, generation modes, and artifact resolution. --- id: benchmark-datasets.cases title: Benchmark Dataset Cases summary: Inspect benchmark Case membership, coverage traces, references, and scoped bulk actions. kind: task product_area: benchmark_datasets status: stable updated: 2026-08-22 canonical: /docs/benchmark-datasets/cases --- # Benchmark Dataset Cases ## Prerequisites - A selected benchmark and permission to inspect or manage its current dataset. - Project Cases that conform to the intended Input Schema. The Cases tab is the benchmark-scoped view of the current editable case set. It shows Case content and membership together with coverage trace, output or reference mapping, and evaluator relationships. Select one or more rows to request an Expert Contribution, create another benchmark from the selection, remove the Cases from the current benchmark, or download them. Removal changes current membership; it does not delete the reusable Case from project Assets or mutate an existing Snapshot. ## Review before snapshotting 1. Confirm each Case still conforms to Project Input Schema and has the intended materials. 2. Inspect Coverage Facet assignments and source or contributor provenance. 3. Check policy and rubric application, including whether eligible evaluator links are approved. 4. Resolve missing or ambiguous output/reference mapping when the workflow requires reference outputs. 5. Use Representation to check whether the set supports the intended claim. > Membership is not evidence yet > > The editable Cases tab can change. Use a Dataset Snapshot and Benchmark Version when an evaluation, comparison, or Improvement Session must remain reproducible. ## After changing membership Open Representation and confirm that the change affected the intended facet or evaluator population. Removing redundant Cases can improve balance even when total Case count falls. Adding many near-duplicates can increase count without adding meaningful coverage. If a selected Case needs content correction, edit it through the owning Case workflow and review every future benchmark that selects it. Existing Snapshots stay unchanged. If the Case reveals an unclear standard, request an Expert Contribution before compensating with more examples. {% example-demo title="Example: scoped bulk action" %} An operator selects five Cases tied to an unresolved exception and requests one Expert Contribution. The Cases remain in the current set while the expert works. After the controlling Rubric is clarified, the team reviews membership and creates a new Snapshot with the approved evaluator links. {% /example-demo %} ## Object and state changes Selected-row removal changes current benchmark membership; creating another benchmark creates a separate benchmark; requesting a Contribution creates scoped expert work. Downloads and inspection are read-only. No action here mutates an existing Snapshot. ## Success criteria - Current membership, Case identity, coverage trace, and evaluator relationships are understood. - Any bulk action affects only the intended selected Cases. - A new Snapshot is created when changed membership must become evaluation evidence. ## Common failure modes - Treating removal from the benchmark as project-level Case deletion. - Assuming editable membership changed an old Benchmark Version. - Selecting Cases by visible text while ignoring their durable IDs. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Benchmark Datasets](/docs/benchmark-datasets) - [Cases](/docs/assets/cases) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Dataset upload](/docs/troubleshooting/dataset-upload) - [Unclear Cases](/docs/troubleshooting/unclear-cases) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} ## Source confidence Code-backed: the active Cases route defines the benchmark membership table, coverage trace, selected-row operations, downloads, and output/reference presentation.