# Case Foundry
Generated: 2026-11-02T06:36:14.512Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: coverage-management.case-foundry
title: Case Foundry
summary: Prepare Case candidates for Coverage Design from current requirements and captured project input settings.
kind: task
product_area: coverage_management
status: stable
updated: 2026-10-30
canonical: /docs/coverage-design/case-foundry
---
# Case Foundry
Case Foundry coordinates Case preparation for the situations defined in Coverage Design. Read preparation progress separately from reviewed Case quality and benchmark membership.
## Prerequisites
- Coverage intent and requirements that identify the missing situation.
- Relevant Story or tuple planning context.
- Project Input Schema and the materials needed by that contract.
- Permission to prepare and review Cases.
## Prepare candidates
Start from Canvas and the intended coverage slice. Confirm whether the work should reuse source material, synthesize a situation or refine existing candidates through the available preparation controls.
Preparation captures the relevant planning and input settings. Its queued, running, partial, completed, failed or cancelled state describes that accepted operation. A completed operation is not a statement that every current requirement is now satisfied.
Project Input Schema owns allowed artifact families, formats and required material keys. An unavailable required family produces an actionable preparation error. Do not replace required material with text or change the project contract merely to make generation appear complete.
## Review the result
Open Case Review to inspect input content, generated materials, provenance and the situation demonstrated by each candidate. Read added, synthesized or retrieved counts as preparation activity. Confirm actual selected membership in Canvas or Benchmark Datasets.
If requirements or source material changed after preparation started, inspect the captured inputs before requesting another operation. Historical completion does not establish freshness against today's settings.
## Object and state changes
Preparation creates an attributable operation and reviewable candidate material. Review and explicit membership operations determine which Cases enter the editable benchmark. Existing Snapshots remain unchanged.
## Success criteria
- Candidate preparation is tied to a named input need.
- The captured contract and actual required materials agree.
- Partial or failed work remains distinguishable from successful output.
- Reviewed membership, rather than generation volume, explains progress.
## Common failure modes
- Generating around an unresolved correctness question.
- Treating operation completion as Case approval.
- Using obsolete Benchmark material preferences instead of Project Input Schema.
- Assuming an old preparation handled newly edited requirements.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Coverage Design](/docs/coverage-design)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Project Input Schema](/docs/project-settings/input-schema)
- [Benchmark Dataset Cases](/docs/benchmark-datasets/cases)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
{% /related-card-grid %}
---
id: coverage-management.overview
title: Coverage Design
summary: Use Canvas to define required input coverage and Inbox to decide how new findings affect benchmark membership.
kind: concept
product_area: coverage_management
status: stable
updated: 2026-10-30
canonical: /docs/coverage-design
---
# Coverage Design
Coverage Design applies reusable project facets to one Benchmark. Use **Canvas** to plan and inspect input coverage, and **Inbox** to decide whether new findings should change the selected Cases.
## Definition
A coverage requirement names an input situation and how it should be represented. It can require a minimum Case count or record an observation. Dimension ontology values, Topics and Scenarios identify the intended slice; distribution settings describe its share of the active benchmark population.
Canvas connects those requirements to actual selected Cases. A large total does not establish that an important combination is present, and unknown classification does not establish absence.
## Decision checkpoint
| Need | Use | Result |
| --- | --- | --- |
| Define the intended behavior space | Canvas requirements and settings | Explicit coverage obligations |
| Inspect a slice or combination | Canvas selection and matrix | Matching Cases, unknowns and remaining need |
| Prepare more Cases | Case preparation | Reviewable candidates and materials |
| Act on an expert or Fieldloop finding | Inbox | An explicit adoption decision |
| Freeze the selected population | Benchmark Datasets → Snapshots | An immutable Benchmark Version |
## Work from Canvas
Describe the Benchmark intent, select the relevant facets and inspect count or distribution requirements. Examine Cases behind a count before deciding that a gap is resolved. Distinguish requirements from observations and current membership from available project Cases.
Stories and their tuples describe meaningful situations used in planning and Case preparation. Case Foundry coordinates preparation; Case Review is where prepared inputs and materials are inspected. These mechanisms support Canvas work rather than replacing its current requirement and membership view.
## Materials and input shape
Project Input Schema owns allowed artifact families, accepted formats and required material keys. Coverage Design does not add another Materials configuration. A tuple can require an allowed artifact but cannot remove a required input or enable an unsupported format.
## Act from Inbox
Inbox links attributable input-coverage findings to possible dataset changes. Review each observation or proposed Case separately. Generating a Case stages another item; it does not automatically add membership. A resolved report can contain declined or withdrawn items.
Existing Snapshots and Runs retain their recorded population. Create a new Snapshot when changed membership or standards should define a later evaluation.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Coverage Design Settings](/docs/coverage-design/settings)
- [Input Coverage Inbox](/docs/coverage-design/inbox)
- [Coverage Stories](/docs/coverage-design/coverage-stories)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Benchmark Datasets](/docs/benchmark-datasets)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
---
id: coverage-management.case-review
title: Review Prepared Cases
summary: Inspect generated inputs and materials against their captured coverage purpose before relying on benchmark membership.
kind: task
product_area: coverage_management
status: stable
updated: 2026-10-30
canonical: /docs/coverage-design/case-review
---
# Review Prepared Cases
Review prepared Cases for the input situation they actually demonstrate. Preparation success, current inclusion and sufficient coverage are separate conclusions.
## Prerequisites
- Prepared candidates and their captured coverage context.
- Permission to manage benchmark membership.
- The Project Input Schema and required materials used for preparation.
## Inspect candidates
Read the canonical input, source relationship, facet assignments and stated fit to the tuple or requirement. Check required files and values rather than assuming a completed generation produced usable materials.
Remove a candidate that is redundant, unsupported or misplaced through the available controls. Inclusion should reflect a deliberate decision about the current editable dataset. It does not change an older Snapshot.
For a tuple that still lacks suitable Cases, generation can append candidates and regeneration can replace the preparation proposal according to its displayed source mode. Preserve whether material was retrieved, reused or synthesized.
## Materials and compatibility
Project Input Schema owns material keys, accepted families and extensions. A required material must be present and usable. Accepted extensions are alternatives, not a demand to generate every listed format.
Generated-material status and Case status may settle at different times. A partial or failed attachment is not a complete Case. Use the reported failure to correct the owning source or contract rather than silently dropping the required material.
## Object and state changes
Review changes candidate inclusion or editable membership through the owning operation. Additional preparation records another captured request. A Snapshot freezes the selected Cases only when explicitly created.
Cases staged by an Inbox generation have a separate adoption decision in Inbox. Completing generation alone does not add those Cases or resolve the source observation.
## Success criteria
- The input demonstrates its intended situation without invented assumptions.
- Required materials are inspectable and conform to the captured contract.
- Current membership matches the review decision.
- Repeated or generated Cases retain their source relationship.
## Common failure modes
- Accepting a candidate because its card exists.
- Using count completion as proof of distinct coverage.
- Confusing generated material with approved correctness.
- Treating a Case correction as a mutation of historical Runs.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Coverage Design](/docs/coverage-design)
- [Input Coverage Inbox](/docs/coverage-design/inbox)
- [Case Foundry](/docs/coverage-design/case-foundry)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
{% /related-card-grid %}
---
id: project-settings.input-schema
title: Project Input Schema
summary: Define the canonical input architecture, case-material fields, and accepted artifact formats for project cases.
kind: reference
product_area: project_settings
status: stable
updated: 2026-10-30
canonical: /docs/project-settings/input-schema
---
# Project Input Schema
## Definition
Project Input Schema is the project-managed contract for future cases. It declares the primary input architecture, optional structured-input schema, named case-material fields, and artifact families or file extensions the project accepts.
The active architectures are **plain text**, **chat**, and **structured**. The schema is a project singleton rather than a versioned benchmark object. When no setting exists, the default accepts one plain-text user message and no case materials or artifacts.
## Fields, states, or lifecycle rules
- `architecture` is `plain_text`, `chat`, or `structured`.
- Structured architecture requires a bounded `structuredInputSchema`.
- `caseMaterialSchema` is a closed, flat object. Each material key has a label, optional description, required flag, type, and any accepted artifact rules.
- Material keys use lowercase letters, digits, and underscores, start with a letter, and remain flat.
- `acceptedArtifacts` declares project-level artifact families. Supported families are image, document, tabular, presentation, source text, and audio.
- A case-material artifact rule must be a subset of the artifact families admitted at the project root.
- Saving a new schema governs future case validation. Operators should inspect existing cases before making a change that would make current content invalid.
## Canonical case content
The primary case payload is `content.input`. Optional supporting values and artifacts live in `content.case_materials`. The product renders `record_content.case_view` so people and execution adapters can inspect the canonical content consistently; that view is a projection rather than an alternate authoring contract.
Static support passed to a Harness uses `case_material_refs`. An executable or queryable environment uses an optional `world_instance_ref`. Do not collapse static documents, images, or values into the world boundary merely because a candidate consumes them during a Run.
{% example-demo title="Example: structured support case" %}
A project selects structured input with `question` and `customer_tier` properties. It declares a required `policy_document` case material that accepts PDF documents and an optional `account_history` tabular material. A case is valid only when its structured input matches the schema and the required document is present in the accepted format.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Project Settings](/docs/project-settings)
- [Product Quickstart](/docs/quickstart)
- [Cases](/docs/assets/cases)
- [Review Screens](/docs/assets/review-screens)
{% /related-card-grid %}
---
id: benchmark-datasets.cases
title: Benchmark Dataset Cases
summary: Inspect benchmark Case membership, coverage traces, references, and scoped bulk actions.
kind: task
product_area: benchmark_datasets
status: stable
updated: 2026-10-30
canonical: /docs/benchmark-datasets/cases
---
# Benchmark Dataset Cases
## Prerequisites
- A selected benchmark and permission to inspect or manage its current dataset.
- Project Cases that conform to the intended Input Schema.
The Cases tab is the benchmark-scoped view of the current editable case set. It shows Case content and membership together with coverage trace, output or reference mapping, and evaluator relationships.
Select one or more rows to request an Expert Contribution, create another benchmark from the selection, remove the Cases from the current benchmark, or download them. Removal changes current membership; it does not delete the reusable Case from project Assets or mutate an existing Snapshot.
## Review before snapshotting
1. Confirm each Case still conforms to Project Input Schema and has the intended materials.
2. Inspect Coverage Facet assignments and source or contributor provenance.
3. Check policy and rubric application, including whether eligible evaluator links are approved.
4. Resolve missing or ambiguous output/reference mapping when the workflow requires reference outputs.
5. Use Representation to check whether the set supports the intended claim.
> Membership is not a frozen evaluation
>
> The editable Cases tab can change. Use a Dataset Snapshot and Benchmark Version when an evaluation, comparison, or Improvement Session must remain reproducible.
## After changing membership
Open Representation and confirm that the change affected the intended facet or evaluator population. Removing redundant Cases can improve balance even when total Case count falls. Adding many near-duplicates can increase count without adding meaningful coverage.
If a selected Case needs content correction, edit it through the owning Case workflow and review every future benchmark that selects it. Existing Snapshots stay unchanged. If the Case reveals an unclear standard, request an Expert Contribution before compensating with more examples.
{% example-demo title="Example: scoped bulk action" %}
An operator selects five Cases tied to an unresolved exception and requests one Expert Contribution. The Cases remain in the current set while the expert works. After the controlling Rubric is clarified, the team reviews membership and creates a new Snapshot with the approved evaluator links.
{% /example-demo %}
## Object and state changes
Selected-row removal changes current benchmark membership; creating another benchmark creates a separate benchmark; requesting a Contribution creates scoped expert work. Downloads and inspection are read-only. No action here mutates an existing Snapshot.
## Success criteria
- Current membership, Case identity, coverage trace, and evaluator relationships are understood.
- Any bulk action affects only the intended selected Cases.
- A new Snapshot is created when changed membership must define a fixed population for future evaluation.
## Common failure modes
- Treating removal from the benchmark as project-level Case deletion.
- Assuming editable membership changed an old Benchmark Version.
- Selecting Cases by visible text while ignoring their durable IDs.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Benchmark Datasets](/docs/benchmark-datasets)
- [Cases](/docs/assets/cases)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Case upload troubleshooting](/docs/troubleshooting/dataset-upload)
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
- [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage)
{% /related-card-grid %}
---
id: troubleshooting.unrealistic-synthetic-cases
title: Synthetic Cases That Feel Unrealistic
summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/unrealistic-synthetic-cases
---
# Synthetic Cases That Feel Unrealistic
## Symptoms
- Generated language sounds artificial or describes a user who would not exist in the product.
- Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere.
- The Case combines incompatible ontology values or violates product constraints.
- Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark.
- Many generated candidates repeat the same template with superficial wording changes.
## Likely causes
- The synthesis request lacks representative source Cases or product constraints.
- A coverage tuple combines values that cannot coexist in the target system.
- A Comparison Direction is too broad, stale, or duplicative.
- Generation is being judged as finished benchmark results instead of candidate material.
## Diagnose the generation boundary
1. Identify the entry surface. **Assets → Cases → Synthesize** creates reusable project Case candidates; **Coverage Design → Case Foundry** coordinates candidates for a named benchmark need.
2. Compare the generated Case with real in-use Cases from the same behavior region.
3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires different inputs or directions.
4. Check whether the generator lacked constraints that operators hold only in private memory.
5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not selected Benchmark Dataset supporting information.
## Fix
- Add representative source Cases or explicit product constraints to the synthesis context.
- Narrow, edit, archive, or delete a Comparison Direction that produces implausible combinations.
- Generate from a specific missing coverage tuple rather than asking for generic diversity.
- Edit a promising candidate when its behavioral boundary is sound and only presentation is weak.
- Reject candidates whose premise is impossible or whose source context cannot be justified.
- Review duplicates before adding any generated Case to a Benchmark.
Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use.
## Prevention
Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Keep Comparison Directions narrow, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot.
{% example-demo title="Example: impossible refund request" %}
Synthesis produces a Case where an anonymous visitor requests an enterprise-contract refund and supplies internal approval metadata in chat. The tuple is valuable—exception request with conflicting authority—but the realization is impossible. The team adds authentication and channel constraints, narrows the Comparison Direction, and regenerates. The realistic candidate still goes through Case review before selection.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Comparison Directions](/docs/assets/comparison-directions)
- [Boundary Cases](/docs/coverage-engineering/boundary-cases)
- [Case Pool](/docs/coverage-engineering/case-pool)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Case Pool surface](/docs/object-model/case-pool)
- [Cases](/docs/object-model/cases)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
{% /related-card-grid %}
---
id: troubleshooting.unclear-cases
title: Unclear Cases
summary: Repair Cases that informed reviewers cannot interpret consistently from the supplied input, context, and output identity.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/unclear-cases
---
# Unclear Cases
## Symptoms
- Reviewers ask what they are supposed to judge.
- The same Case produces incompatible readings because a source, prior turn, time boundary, or user state is absent.
- A candidate answer is visible, but it is unclear whether it is the judged output or reference behavior.
- The Case title or metadata implies context that the review presentation does not contain.
- Experts can answer only by inventing assumptions.
## Likely causes
- A prior conversation turn, source, time boundary, or user state is missing.
- Candidate and reference outputs are swapped or unlabeled.
- One Case combines multiple situations or requested judgments.
- Important instructions live in metadata or private project knowledge instead of visible context.
## Diagnose the Case, not the reviewer
1. Open the Case in its normal review presentation and read only what the assigned expert can see.
2. Identify the judged input and output separately. For conversations, confirm the turn order and which response is under review.
3. List every fact required to make the judgment, then verify each fact is present as Case content, context, source material, or an explicitly linked standard.
4. Check whether metadata is being used as hidden instruction rather than visible context.
5. Compare reviewer rationale. Repeated invented assumptions usually reveal the missing boundary.
## Fix
- Add or map the missing context when it belongs to the real situation.
- Correct the output attachment when candidate and reference behavior were swapped.
- Split a Case that combines two independently judged situations.
- Narrow the question or applicability boundary when multiple interpretations are legitimate.
- Mark the Case unresolved or keep it as candidate material when authoritative supporting information is unavailable.
- Create a new Case version when the reviewable content changes; do not silently rewrite supporting information used by an existing Snapshot.
After correction, have a reviewer who did not author the Case explain the situation and requested judgment. If they must rely on private project knowledge, the Case is still incomplete.
## Prevention
Review the Case through the same presentation an expert will see, label output identity explicitly, and include every fact required for judgment. Sample imported and generated Cases before Benchmark selection, and version content changes after supporting information exists.
{% example-demo title="Example: recommendation without the user's equipment" %}
A Case asks whether a product is compatible but omits the equipment model mentioned in an earlier conversation turn. Reviewers alternate between pass and fail. The operator restores the prior turn, confirms the candidate output identity, versions the Case, and repeats the affected review. The old Snapshot retains the earlier ambiguous version.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Assets](/docs/assets)
- [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution)
- [Boundary Cases](/docs/coverage-engineering/boundary-cases)
- [Case Versioning](/docs/governance/case-versioning)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Cases](/docs/object-model/cases)
- [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases)
- [Metadata and context](/docs/reference/metadata-and-context)
{% /related-card-grid %}