# Coverage Design Generated: 2026-11-03T06:59:26.242Z Source build: local Canonical docs: https://teammately.ai/docs --- id: coverage-management.overview title: Coverage Design summary: Use Canvas to define required input coverage and Inbox to decide how new findings affect benchmark membership. kind: concept product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design --- # Coverage Design Coverage Design applies reusable project facets to one Benchmark. Use **Canvas** to plan and inspect input coverage, and **Inbox** to decide whether new findings should change the selected Cases. ## Definition A coverage requirement names an input situation and how it should be represented. It can require a minimum Case count or record an observation. Dimension ontology values, Topics and Scenarios identify the intended slice; distribution settings describe its share of the active benchmark population. Canvas connects those requirements to actual selected Cases. A large total does not establish that an important combination is present, and unknown classification does not establish absence. ## Decision checkpoint | Need | Use | Result | | --- | --- | --- | | Define the intended behavior space | Canvas requirements and settings | Explicit coverage obligations | | Inspect a slice or combination | Canvas selection and matrix | Matching Cases, unknowns and remaining need | | Prepare more Cases | Case preparation | Reviewable candidates and materials | | Act on an expert or Fieldloop finding | Inbox | An explicit adoption decision | | Freeze the selected population | Benchmark Datasets → Snapshots | An immutable Benchmark Version | ## Work from Canvas Describe the Benchmark intent, select the relevant facets and inspect count or distribution requirements. Examine Cases behind a count before deciding that a gap is resolved. Distinguish requirements from observations and current membership from available project Cases. Stories and their tuples describe meaningful situations used in planning and Case preparation. Case Foundry coordinates preparation; Case Review is where prepared inputs and materials are inspected. These mechanisms support Canvas work rather than replacing its current requirement and membership view. ## Materials and input shape Project Input Schema owns allowed artifact families, accepted formats and required material keys. Coverage Design does not add another Materials configuration. A tuple can require an allowed artifact but cannot remove a required input or enable an unsupported format. ## Act from Inbox Inbox links attributable input-coverage findings to possible dataset changes. Review each observation or proposed Case separately. Generating a Case stages another item; it does not automatically add membership. A resolved report can contain declined or withdrawn items. Existing Snapshots and Runs retain their recorded population. Create a new Snapshot when changed membership or standards should define a later evaluation. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design Settings](/docs/coverage-design/settings) - [Input Coverage Inbox](/docs/coverage-design/inbox) - [Coverage Stories](/docs/coverage-design/coverage-stories) - [Review Prepared Cases](/docs/coverage-design/case-review) - [Benchmark Datasets](/docs/benchmark-datasets) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} --- id: coverage-management.case-foundry title: Case Foundry summary: Prepare Case candidates for Coverage Design from current requirements and captured project input settings. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/case-foundry --- # Case Foundry Case Foundry coordinates Case preparation for the situations defined in Coverage Design. Read preparation progress separately from reviewed Case quality and benchmark membership. ## Prerequisites - Coverage intent and requirements that identify the missing situation. - Relevant Story or tuple planning context. - Project Input Schema and the materials needed by that contract. - Permission to prepare and review Cases. ## Prepare candidates Start from Canvas and the intended coverage slice. Confirm whether the work should reuse source material, synthesize a situation or refine existing candidates through the available preparation controls. Preparation captures the relevant planning and input settings. Its queued, running, partial, completed, failed or cancelled state describes that accepted operation. A completed operation is not a statement that every current requirement is now satisfied. Project Input Schema owns allowed artifact families, formats and required material keys. An unavailable required family produces an actionable preparation error. Do not replace required material with text or change the project contract merely to make generation appear complete. ## Review the result Open Case Review to inspect input content, generated materials, provenance and the situation demonstrated by each candidate. Read added, synthesized or retrieved counts as preparation activity. Confirm actual selected membership in Canvas or Benchmark Datasets. If requirements or source material changed after preparation started, inspect the captured inputs before requesting another operation. Historical completion does not establish freshness against today's settings. ## Object and state changes Preparation creates an attributable operation and reviewable candidate material. Review and explicit membership operations determine which Cases enter the editable benchmark. Existing Snapshots remain unchanged. ## Success criteria - Candidate preparation is tied to a named input need. - The captured contract and actual required materials agree. - Partial or failed work remains distinguishable from successful output. - Reviewed membership, rather than generation volume, explains progress. ## Common failure modes - Generating around an unresolved correctness question. - Treating operation completion as Case approval. - Using obsolete Benchmark material preferences instead of Project Input Schema. - Assuming an old preparation handled newly edited requirements. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Review Prepared Cases](/docs/coverage-design/case-review) - [Project Input Schema](/docs/project-settings/input-schema) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) - [Unclear Cases](/docs/troubleshooting/unclear-cases) {% /related-card-grid %} --- id: coverage-management.case-review title: Review Prepared Cases summary: Inspect generated inputs and materials against their captured coverage purpose before relying on benchmark membership. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/case-review --- # Review Prepared Cases Review prepared Cases for the input situation they actually demonstrate. Preparation success, current inclusion and sufficient coverage are separate conclusions. ## Prerequisites - Prepared candidates and their captured coverage context. - Permission to manage benchmark membership. - The Project Input Schema and required materials used for preparation. ## Inspect candidates Read the canonical input, source relationship, facet assignments and stated fit to the tuple or requirement. Check required files and values rather than assuming a completed generation produced usable materials. Remove a candidate that is redundant, unsupported or misplaced through the available controls. Inclusion should reflect a deliberate decision about the current editable dataset. It does not change an older Snapshot. For a tuple that still lacks suitable Cases, generation can append candidates and regeneration can replace the preparation proposal according to its displayed source mode. Preserve whether material was retrieved, reused or synthesized. ## Materials and compatibility Project Input Schema owns material keys, accepted families and extensions. A required material must be present and usable. Accepted extensions are alternatives, not a demand to generate every listed format. Generated-material status and Case status may settle at different times. A partial or failed attachment is not a complete Case. Use the reported failure to correct the owning source or contract rather than silently dropping the required material. ## Object and state changes Review changes candidate inclusion or editable membership through the owning operation. Additional preparation records another captured request. A Snapshot freezes the selected Cases only when explicitly created. Cases staged by an Inbox generation have a separate adoption decision in Inbox. Completing generation alone does not add those Cases or resolve the source observation. ## Success criteria - The input demonstrates its intended situation without invented assumptions. - Required materials are inspectable and conform to the captured contract. - Current membership matches the review decision. - Repeated or generated Cases retain their source relationship. ## Common failure modes - Accepting a candidate because its card exists. - Using count completion as proof of distinct coverage. - Confusing generated material with approved correctness. - Treating a Case correction as a mutation of historical Runs. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Input Coverage Inbox](/docs/coverage-design/inbox) - [Case Foundry](/docs/coverage-design/case-foundry) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unclear Cases](/docs/troubleshooting/unclear-cases) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: coverage-management.coverage-stories title: Coverage Stories summary: Describe meaningful input situations and testable facet combinations used in Coverage Design and Case preparation. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/coverage-stories --- # Coverage Stories Stories explain the situation behind a coverage requirement. They connect Benchmark intent to testable facet combinations and the Cases prepared to exercise them. ## Prerequisites - An identified Benchmark intent and current coverage requirements. - Dimensions, Topics or Scenarios that describe the relevant distinctions. - A Case input contract that can represent the situation. ## Make a Story concrete Describe the actor, request, source condition, conflict or constraint that changes expected behavior. A Story should help an operator explain why a slice matters without treating a chart category as a complete situation. Tuples name smaller obligations. Their statement of what must be demonstrated, facet references and intended count guide preparation and review. A tuple's requested count does not prove that its Cases are realistic or distinct. ## Read Stories with Canvas Use Canvas to inspect current requirements and selected Cases. Stories supply planning context; requirement assessments supply current matched, unknown and remaining counts. Do not substitute a Story budget for actual dataset membership. Where a Story is created or edited through the available controls, preserve its rationale and relationship to the relevant facets. An archived Story is history rather than a new instruction to remove every related Case. ## Materials Project Input Schema supplies allowed artifacts and required material keys. Story planning can request an appropriate allowed artifact. It cannot disable required input, add an unaccepted family or create another materials policy. ## Object and state changes Story edits change current planning context. Prepared work records the Story and input contract it accepted. Existing Snapshots and completed Runs retain their original Cases and criteria. ## Success criteria - A reviewer can tell what situation each tuple must demonstrate. - Facet references identify the intended project concepts. - Counts are interpreted alongside actual Case quality and membership. - Current requirements and captured preparation inputs are distinguished. ## Common failure modes - Using a generic title as the entire requirement. - Counting near-duplicate Cases as distinct coverage. - Reconstructing an accepted preparation from today's edited Story. - Treating Story generation as automatic Case adoption. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Coverage Design Settings](/docs/coverage-design/settings) - [Case Foundry](/docs/coverage-design/case-foundry) - [Review Prepared Cases](/docs/coverage-design/case-review) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: coverage-design.inbox title: Input Coverage Inbox summary: Review attributable input-coverage findings and decide whether to generate, add, remove, or decline Cases. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/inbox --- # Input Coverage Inbox Use **Coverage Design → Inbox** when an expert or Fieldloop identifies an input situation that may change the Benchmark's selected Cases. ## Prerequisites - A settled source report delivered to the selected Benchmark. - Permission to manage its Cases. - Enough source context to decide whether the input need is relevant. ## Read the source Contribution reports contain effective expert-confirmed coverage observations and attributable newly synthesized Input Cases after their completion and checkpoint reconciliation. Settled results can also be retained from a cancelled Contribution. Fieldloop reports contain individually identified input-coverage discoveries tied to published sources and Benchmark references. A correctness-only report or an arbitrary paragraph does not become an Inbox item. Lemon Goal reports remain in their own feature. ## Decide each item A proposed new Case supports **Add to benchmark** or **Do not adopt**. An observation can lead to generating a new Case, adding existing Cases, removing current members, recording that no dataset change is needed, or declining adoption. Generation requests one new input under the project contract and stages it as a separate item. Inspect that Case and decide adoption. A failed generation or zero returned Cases leaves the observation open. ## Object and state changes Adoption changes editable membership and records the decision together. Retries preserve the original outcome. An already-present Case is not added twice. Items can be pending, in progress, applied, dismissed or withdrawn. Reports show **Needs review** while effective unresolved items remain and **Resolved** when none remain. Resolved does not necessarily mean the dataset changed: declined or withdrawn items can also be settled. Later attributable results can arrive in the same report. Corrected observations remain distinguishable from an older decision. A dismissed item is not reopened merely by rediscovery. ## Success criteria - Each action refers to the current source item. - Generated Cases receive a separate adoption decision. - Actual membership, rather than report viewing or generation completion, determines whether the dataset changed. - Historical Snapshots remain unchanged. ## Common failure modes - Treating a correctness concern as input coverage automatically. - Assuming generation adds a Case. - Interpreting Resolved as universal acceptance. - Repeating a settled decision because a report was refreshed. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Contributed Artifacts](/docs/expert-contributions/contributed-artifacts) - [Fieldloop](/docs/fieldloop) - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) - [Expert Contribution problems](/docs/troubleshooting/expert-contributions) {% /related-card-grid %} --- id: coverage-management.get-started title: Coverage Design Settings summary: Set benchmark intent, facet handling, and coverage requirements while inheriting materials from Project Input Schema. kind: task product_area: coverage_management status: stable updated: 2026-10-30 canonical: /docs/coverage-design/settings --- # Coverage Design Settings Use the settings and requirement controls in **Coverage Design → Canvas** to state which input situations the Benchmark must represent. ## Prerequisites - A selected Benchmark and permission to manage it. - Project facets that describe the intended distinctions. - A saved Project Input Schema. - A concrete statement of Benchmark intent. ## Define intent and facet handling Explain the intended behavior and the boundaries that matter. Choose Dimension ontology roles, Topic Group handling and Scenario steering for their different purposes. A required ontology value identifies a coverage obligation. A Topic Group can require every member Topic, require group-level representation, provide guidance or be excluded. Scenario steering controls which reusable constructions are preferred, avoided or left to the system. These controls do not make every active project facet mandatory. ## Define requirements Name each situation and its facets. Use a minimum Case count when a specific volume is required. Distribution bounds or targets express the desired share of the active benchmark population. Read matched, unknown and remaining counts separately. An unknown classification is not a negative match. Inspect the actual selected Cases before concluding that a requirement is met. ## Inherit materials Allowed artifacts, file formats and required Case materials come from **Project Input Schema**. Change them there when the project contract needs to change. Canvas has no separate Materials settings. A required image material remains required even if an old Benchmark setting had excluded images. Accepting PNG and JPEG permits alternatives; it does not require both. New preparation captures the effective schema-derived contract. Accepted earlier work retains its captured settings. ## Object and state changes Saving changes current benchmark coverage intent or requirements. It does not approve a generated Case, change an existing Snapshot or rewrite accepted preparation inputs. Actual Case adoption changes editable membership through its owning action. ## Success criteria - Requirements describe meaningful situations and the intended population. - Count and distribution expectations are explicit. - Unknown classifications remain visible. - Materials agree with the project contract. ## Common failure modes - Treating a Topic Group as though it merges its Topics. - Requiring every active facet without a reason. - Attempting to override required materials at Benchmark scope. - Declaring a gap closed from a total count without inspecting the Cases. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Coverage Design](/docs/coverage-design) - [Dimensions and Ontology](/docs/coverage-engineering/dimensions-ontology) - [Scenarios](/docs/coverage-engineering/scenarios) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: benchmark-datasets.overview title: Benchmark Datasets summary: Select benchmark Cases, inspect representation, and freeze immutable Snapshots for reproducible supporting information. kind: concept product_area: benchmark_datasets status: stable updated: 2026-10-30 canonical: /docs/benchmark-datasets --- # Benchmark Datasets Benchmark Datasets defines the evaluation set for one benchmark through **Cases**, **Representation**, and **Snapshots**. The current dataset is editable. It selects reusable project Cases and reflects current facet, policy, rubric, and contributor facts. A Snapshot freezes the exact dataset state needed by a Benchmark Version and its evaluations. These are deliberately different surfaces: editing the current set must not rewrite historical results. ## Decision checkpoint | Surface | Use it to | Supporting information rule | | --- | --- | --- | | Cases | Inspect and change current benchmark membership | Selection is live until snapshotted | | Representation | Find concentration and absence across governed facets | Read distribution together with distinct Case counts | | Snapshots | Freeze Cases, evaluator links, and representation facts | Snapshot content is read-only | Coverage Design acts on gaps found in the dataset. Assets remains the project-level reusable pool. Benchmark Evaluations runs exact Harness Versions against an immutable Benchmark Version rather than an unspecified “current dataset.” ## Supporting information flow Cases usually begin in project Assets or materialize through Case Review or Expert Contributions. Selecting them makes them part of the current benchmark dataset. Representation then summarizes the current assignments and evaluator relationships. Snapshot readiness checks whether that state can be frozen. A Snapshot supplies the immutable dataset facts used by a Benchmark Version. This flow is one-way for historical results. Later edits to an Asset, facet assignment, Policy, Rubric, or current membership may improve the next Snapshot, but they do not update a previous Snapshot. Compare candidates within one Benchmark Version unless the analysis explicitly accounts for a moved snapshot boundary. ## Before creating a Snapshot Check Case clarity and schema conformance, then inspect Representation for intended behavior and provenance. Confirm approved eligible evaluator links. Resolve Snapshot blockers and preserve the resulting label, version, content hash, creation time, and Case count. A Snapshot can be reproducible while still being incomplete as product coverage. A Snapshot identifies fixed inputs and criteria. Representation and Coverage Design help assess whether that input population supports the intended coverage claim; actual evaluation results come from a Run. {% example-demo title="Example: editable set versus frozen inputs and criteria" %} The current dataset gains four Cases and a corrected Rubric link after an expert Contribution is reconciled. An earlier Run still points to its old Benchmark Version. The operator creates a new Snapshot and Version for the changed set rather than comparing the new candidate against the old Run as though only Harness behavior moved. {% /example-demo %} {% related-card-grid title="Dataset workflows" %} - [Benchmark Dataset Cases](/docs/benchmark-datasets/cases) - [Dataset Representation](/docs/benchmark-datasets/representation) - [Dataset Snapshots](/docs/benchmark-datasets/snapshots) - [Coverage Design](/docs/coverage-design) {% /related-card-grid %} --- id: project-settings.input-schema title: Project Input Schema summary: Define the canonical input architecture, case-material fields, and accepted artifact formats for project cases. kind: reference product_area: project_settings status: stable updated: 2026-10-30 canonical: /docs/project-settings/input-schema --- # Project Input Schema ## Definition Project Input Schema is the project-managed contract for future cases. It declares the primary input architecture, optional structured-input schema, named case-material fields, and artifact families or file extensions the project accepts. The active architectures are **plain text**, **chat**, and **structured**. The schema is a project singleton rather than a versioned benchmark object. When no setting exists, the default accepts one plain-text user message and no case materials or artifacts. ## Fields, states, or lifecycle rules - `architecture` is `plain_text`, `chat`, or `structured`. - Structured architecture requires a bounded `structuredInputSchema`. - `caseMaterialSchema` is a closed, flat object. Each material key has a label, optional description, required flag, type, and any accepted artifact rules. - Material keys use lowercase letters, digits, and underscores, start with a letter, and remain flat. - `acceptedArtifacts` declares project-level artifact families. Supported families are image, document, tabular, presentation, source text, and audio. - A case-material artifact rule must be a subset of the artifact families admitted at the project root. - Saving a new schema governs future case validation. Operators should inspect existing cases before making a change that would make current content invalid. ## Canonical case content The primary case payload is `content.input`. Optional supporting values and artifacts live in `content.case_materials`. The product renders `record_content.case_view` so people and execution adapters can inspect the canonical content consistently; that view is a projection rather than an alternate authoring contract. Static support passed to a Harness uses `case_material_refs`. An executable or queryable environment uses an optional `world_instance_ref`. Do not collapse static documents, images, or values into the world boundary merely because a candidate consumes them during a Run. {% example-demo title="Example: structured support case" %} A project selects structured input with `question` and `customer_tier` properties. It declares a required `policy_document` case material that accepts PDF documents and an optional `account_history` tabular material. A case is valid only when its structured input matches the schema and the required document is present in the accepted format. {% /example-demo %} ## Related task pages {% related-card-grid title="Related task pages" %} - [Project Settings](/docs/project-settings) - [Product Quickstart](/docs/quickstart) - [Cases](/docs/assets/cases) - [Review Screens](/docs/assets/review-screens) {% /related-card-grid %}