# Coverage Design Settings
Generated: 2026-11-03T08:05:53.938Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: coverage-management.get-started
title: Coverage Design Settings
summary: Set benchmark intent, facet handling, and coverage requirements while inheriting materials from Project Input Schema.
kind: task
product_area: coverage_management
status: stable
updated: 2026-10-30
canonical: /docs/coverage-design/settings
---
# Coverage Design Settings
Use the settings and requirement controls in **Coverage Design → Canvas** to state which input situations the Benchmark must represent.
## Prerequisites
- A selected Benchmark and permission to manage it.
- Project facets that describe the intended distinctions.
- A saved Project Input Schema.
- A concrete statement of Benchmark intent.
## Define intent and facet handling
Explain the intended behavior and the boundaries that matter. Choose Dimension ontology roles, Topic Group handling and Scenario steering for their different purposes.
A required ontology value identifies a coverage obligation. A Topic Group can require every member Topic, require group-level representation, provide guidance or be excluded. Scenario steering controls which reusable constructions are preferred, avoided or left to the system. These controls do not make every active project facet mandatory.
## Define requirements
Name each situation and its facets. Use a minimum Case count when a specific volume is required. Distribution bounds or targets express the desired share of the active benchmark population.
Read matched, unknown and remaining counts separately. An unknown classification is not a negative match. Inspect the actual selected Cases before concluding that a requirement is met.
## Inherit materials
Allowed artifacts, file formats and required Case materials come from **Project Input Schema**. Change them there when the project contract needs to change. Canvas has no separate Materials settings.
A required image material remains required even if an old Benchmark setting had excluded images. Accepting PNG and JPEG permits alternatives; it does not require both. New preparation captures the effective schema-derived contract. Accepted earlier work retains its captured settings.
## Object and state changes
Saving changes current benchmark coverage intent or requirements. It does not approve a generated Case, change an existing Snapshot or rewrite accepted preparation inputs. Actual Case adoption changes editable membership through its owning action.
## Success criteria
- Requirements describe meaningful situations and the intended population.
- Count and distribution expectations are explicit.
- Unknown classifications remain visible.
- Materials agree with the project contract.
## Common failure modes
- Treating a Topic Group as though it merges its Topics.
- Requiring every active facet without a reason.
- Attempting to override required materials at Benchmark scope.
- Declaring a gap closed from a total count without inspecting the Cases.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Coverage Design](/docs/coverage-design)
- [Dimensions and Ontology](/docs/coverage-engineering/dimensions-ontology)
- [Scenarios](/docs/coverage-engineering/scenarios)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage)
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
{% /related-card-grid %}
---
id: coverage-management.overview
title: Coverage Design
summary: Use Canvas to define required input coverage and Inbox to decide how new findings affect benchmark membership.
kind: concept
product_area: coverage_management
status: stable
updated: 2026-10-30
canonical: /docs/coverage-design
---
# Coverage Design
Coverage Design applies reusable project facets to one Benchmark. Use **Canvas** to plan and inspect input coverage, and **Inbox** to decide whether new findings should change the selected Cases.
## Definition
A coverage requirement names an input situation and how it should be represented. It can require a minimum Case count or record an observation. Dimension ontology values, Topics and Scenarios identify the intended slice; distribution settings describe its share of the active benchmark population.
Canvas connects those requirements to actual selected Cases. A large total does not establish that an important combination is present, and unknown classification does not establish absence.
## Decision checkpoint
| Need | Use | Result |
| --- | --- | --- |
| Define the intended behavior space | Canvas requirements and settings | Explicit coverage obligations |
| Inspect a slice or combination | Canvas selection and matrix | Matching Cases, unknowns and remaining need |
| Prepare more Cases | Case preparation | Reviewable candidates and materials |
| Act on an expert or Fieldloop finding | Inbox | An explicit adoption decision |
| Freeze the selected population | Benchmark Datasets → Snapshots | An immutable Benchmark Version |
## Work from Canvas
Describe the Benchmark intent, select the relevant facets and inspect count or distribution requirements. Examine Cases behind a count before deciding that a gap is resolved. Distinguish requirements from observations and current membership from available project Cases.
Stories and their tuples describe meaningful situations used in planning and Case preparation. Case Foundry coordinates preparation; Case Review is where prepared inputs and materials are inspected. These mechanisms support Canvas work rather than replacing its current requirement and membership view.
## Materials and input shape
Project Input Schema owns allowed artifact families, accepted formats and required material keys. Coverage Design does not add another Materials configuration. A tuple can require an allowed artifact but cannot remove a required input or enable an unsupported format.
## Act from Inbox
Inbox links attributable input-coverage findings to possible dataset changes. Review each observation or proposed Case separately. Generating a Case stages another item; it does not automatically add membership. A resolved report can contain declined or withdrawn items.
Existing Snapshots and Runs retain their recorded population. Create a new Snapshot when changed membership or standards should define a later evaluation.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Coverage Design Settings](/docs/coverage-design/settings)
- [Input Coverage Inbox](/docs/coverage-design/inbox)
- [Coverage Stories](/docs/coverage-design/coverage-stories)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Benchmark Datasets](/docs/benchmark-datasets)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
---
id: coverage.dimensions-ontology
title: Dimensions and Ontology
summary: Define reusable behavior axes and their allowed values, then inspect how cases and benchmarks cover them.
kind: reference
product_area: coverage_engineering
status: stable
updated: 2026-10-30
canonical: /docs/coverage-engineering/dimensions-ontology
---
# Dimensions and Ontology
## Definition
Dimensions are reusable project-level axes for describing how cases differ. Each Dimension contains ontology values: the named members used to classify cases and measure representation. A Dimension might be **Source condition**, with values such as **Current**, **Superseded**, **Conflicting**, and **Missing**.
Use **Project Foundations → Coverage Facets → Dimension** to create, generate, inspect, and maintain them.
## Fields, states, or lifecycle rules
### What a Dimension contains
| Element | Purpose |
| --- | --- |
| Name and description | Explain the behavior axis and its boundary |
| Ontology values | Define the values used for classification |
| Examples | Show classified cases and the reason for a value assignment |
| Statistics | Show case-pool and benchmark distribution by ontology value |
| Benchmark focus | Show whether values are required, sampled, diagnostic, or ignored in benchmark setup |
Dimensions and ontology values are project foundations. A benchmark does not copy them. **Canvas settings** assigns benchmark-specific roles to the project values, and **Representation** reports how the selected cases cover them.
Use the table to compare each Dimension's definition with its ontology and current classification footprint before opening the detail view.
### Create or generate a schema
Create a Dimension manually when the axis and vocabulary are already understood. Use the dimension-schema generator when project context or source material should produce a reviewable proposal. Generated proposals can include a definition, why the Dimension matters, proposed ontology members, and warnings.
A proposal is not the active schema. Review each proposed Dimension and value before accepting it. Avoid accepting near-duplicates simply because they use different wording.
> Classification boundary
>
> Creating or editing a Dimension does not instantly classify every existing case. Missing or stale classifications can be queued and monitored separately. Treat unclassified cases as missing supporting information, not as an implicit ontology value.
### Design rules
- Make the Dimension answer one stable question. Split axes that mix several independent concerns.
- Give every ontology value a definition that distinguishes it from neighboring values.
- Prefer values that can be applied consistently to real cases.
- Do not use a Dimension to encode case quality, policy approval, or a desired model score.
- Review distributions after changing values. A clean schema can still leave important cases unclassified.
- Delete only after checking Case Pool and benchmark usage; removal changes the project coverage vocabulary.
### Benchmark roles
In benchmark Canvas settings, each Dimension and ontology value can receive a focus role:
- **Required:** the benchmark is expected to cover this value.
- **Sampled:** include it as part of the desired mix.
- **Diagnostic:** track it for analysis without making it part of the main denominator.
- **Ignored:** exclude it from the benchmark coverage intention.
- **Unset:** no explicit benchmark instruction has been recorded.
Those roles shape Coverage Story generation and interpretation. They do not alter the project-level definition of the value.
{% example-demo title="Example: source condition" %}
The project defines a Source condition Dimension with Current, Superseded, Conflicting, and Missing values. One benchmark marks all four as required; another marks Current as required and the remaining values as diagnostic. The same project vocabulary supports different benchmark intentions without duplicating the Dimension.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Generate a dimension schema](/docs/coverage-engineering/generate-dimension-schema)
- [Topics](/docs/coverage-engineering/project-topics)
- [Plan Benchmark Coverage](/docs/coverage-engineering/plan-benchmark-coverage)
- [Dataset Representation](/docs/benchmark-datasets/representation)
{% /related-card-grid %}
---
id: coverage.scenarios
title: Scenarios
summary: Define reusable mechanisms for constructing cases and steer how benchmarks use or avoid them.
kind: reference
product_area: coverage_engineering
status: stable
updated: 2026-10-30
canonical: /docs/coverage-engineering/scenarios
---
# Scenarios
## Definition
A Scenario describes a reusable mechanism for building cases. It answers “how should this situation be constructed?” while Dimensions describe differentiating values and Topics describe subject matter.
Examples include conflicting authorities, missing prerequisite supporting information, ambiguous user intent, multi-step state change, or a plausible but superseded source. A good Scenario is portable across Topics rather than tied to one case's wording.
## Fields, states, or lifecycle rules
Each Scenario has a name, description, origin, usage counts, examples, and benchmark statistics. Origins currently distinguish manual, AI-generated, expert-input, and imported Scenarios. Usage can show Case Pool cases, benchmark cases, requirements, and benchmark steering.
Scenario suggestions can be grounded in Topics, source material, existing cases and Scenarios, the Project Agent Brief, Dimensions and ontology, or expert input. Generated candidates include the proposed definition, why they were suggested, and source references. Accept or reject each candidate explicitly.
## Benchmark steering
Canvas settings can use one of three modes:
- **System choose:** allow construction to select appropriate Scenarios from the project library.
- **Prefer:** encourage named Scenarios when building Coverage Stories and cases.
- **Avoid:** prevent named Scenarios from shaping this benchmark's intended case supply.
Coverage Story tuples can also reference specific Scenarios. The tuple reference is the concrete construction instruction for that part of the story; the project Scenario remains reusable and independently editable.
## Design rules
- Name the mechanism, not the domain example. “Conflicting authorities” travels better than “Conflicting HR policies.”
- Explain what makes a case instantiate the Scenario.
- Keep Topics out of the Scenario definition unless they are only examples.
- Do not encode an expected answer or rubric verdict as a scenario.
- Inspect examples and usage before deleting or materially changing a Scenario.
- Treat AI and expert suggestions as reviewable proposals. Origin is provenance, not approval.
> Scenario versus requirement
>
> A Scenario describes a reusable structure. Benchmark requirements describe constraints for one benchmark, and a Coverage Tuple combines the relevant Dimensions, Topic Groups, Scenarios, proof statement, and target case count.
{% example-demo title="Example: portable scenario" %}
The project defines **Plausible superseded authority**: construct a case where an older source appears credible but a newer source controls. Coverage Stories reuse the Scenario for procurement, support, and compliance Topics while applying different source-condition and impact ontology values.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Topics](/docs/coverage-engineering/project-topics)
- [Coverage Design Settings](/docs/coverage-design/settings)
- [Coverage Stories](/docs/coverage-design/coverage-stories)
- [Review Prepared Cases](/docs/coverage-design/case-review)
{% /related-card-grid %}
---
id: project-settings.input-schema
title: Project Input Schema
summary: Define the canonical input architecture, case-material fields, and accepted artifact formats for project cases.
kind: reference
product_area: project_settings
status: stable
updated: 2026-10-30
canonical: /docs/project-settings/input-schema
---
# Project Input Schema
## Definition
Project Input Schema is the project-managed contract for future cases. It declares the primary input architecture, optional structured-input schema, named case-material fields, and artifact families or file extensions the project accepts.
The active architectures are **plain text**, **chat**, and **structured**. The schema is a project singleton rather than a versioned benchmark object. When no setting exists, the default accepts one plain-text user message and no case materials or artifacts.
## Fields, states, or lifecycle rules
- `architecture` is `plain_text`, `chat`, or `structured`.
- Structured architecture requires a bounded `structuredInputSchema`.
- `caseMaterialSchema` is a closed, flat object. Each material key has a label, optional description, required flag, type, and any accepted artifact rules.
- Material keys use lowercase letters, digits, and underscores, start with a letter, and remain flat.
- `acceptedArtifacts` declares project-level artifact families. Supported families are image, document, tabular, presentation, source text, and audio.
- A case-material artifact rule must be a subset of the artifact families admitted at the project root.
- Saving a new schema governs future case validation. Operators should inspect existing cases before making a change that would make current content invalid.
## Canonical case content
The primary case payload is `content.input`. Optional supporting values and artifacts live in `content.case_materials`. The product renders `record_content.case_view` so people and execution adapters can inspect the canonical content consistently; that view is a projection rather than an alternate authoring contract.
Static support passed to a Harness uses `case_material_refs`. An executable or queryable environment uses an optional `world_instance_ref`. Do not collapse static documents, images, or values into the world boundary merely because a candidate consumes them during a Run.
{% example-demo title="Example: structured support case" %}
A project selects structured input with `question` and `customer_tier` properties. It declares a required `policy_document` case material that accepts PDF documents and an optional `account_history` tabular material. A case is valid only when its structured input matches the schema and the required document is present in the accepted format.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Project Settings](/docs/project-settings)
- [Product Quickstart](/docs/quickstart)
- [Cases](/docs/assets/cases)
- [Review Screens](/docs/assets/review-screens)
{% /related-card-grid %}
---
id: troubleshooting.unbalanced-coverage
title: Unbalanced Coverage
summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/unbalanced-coverage
---
# Unbalanced Coverage
Use this when benchmark results overrepresents some behavior and misses other behavior that matters before human review.
## Symptom
A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent.
## Likely causes
- Case import mirrored production volume instead of product risk.
- Dimensions or metadata do not represent the behavior areas that matter.
- Synthetic cases were added in one area but not reviewed across the full risk boundary.
- Coverage was not refreshed after new signals, policy changes, or failures.
## Diagnostic checks
- Inspect **Benchmark Datasets → Representation** and the Coverage Design overview by the configured Coverage Facets.
- Look for policies or rubrics with very few applicable cases.
- Compare production or expert signals with benchmark case distribution.
- Inspect whether new cases are candidate-only or actually in use.
## Fix
- Add representative, boundary, or adversarial cases for undercovered areas.
- Refresh dimensions or metadata before judging balance.
- Review and promote candidate cases into the benchmark only after context is complete.
- Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance.
## Prevention
- Define coverage targets before benchmark runs.
- Track candidate and in-use cases separately.
- Review coverage after policy, rubric, or product behavior changes.
- Do not use aggregate scores without a coverage note.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Dataset Representation](/docs/benchmark-datasets/representation)
- [Coverage Design Settings](/docs/coverage-design/settings)
- [Coverage Stories](/docs/coverage-design/coverage-stories)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Case Pool surface](/docs/object-model/case-pool)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
- [Benchmarks](/docs/object-model/benchmarks)
{% /related-card-grid %}
---
id: troubleshooting.unrealistic-synthetic-cases
title: Synthetic Cases That Feel Unrealistic
summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/unrealistic-synthetic-cases
---
# Synthetic Cases That Feel Unrealistic
## Symptoms
- Generated language sounds artificial or describes a user who would not exist in the product.
- Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere.
- The Case combines incompatible ontology values or violates product constraints.
- Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark.
- Many generated candidates repeat the same template with superficial wording changes.
## Likely causes
- The synthesis request lacks representative source Cases or product constraints.
- A coverage tuple combines values that cannot coexist in the target system.
- A Comparison Direction is too broad, stale, or duplicative.
- Generation is being judged as finished benchmark results instead of candidate material.
## Diagnose the generation boundary
1. Identify the entry surface. **Assets → Cases → Synthesize** creates reusable project Case candidates; **Coverage Design → Case Foundry** coordinates candidates for a named benchmark need.
2. Compare the generated Case with real in-use Cases from the same behavior region.
3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires different inputs or directions.
4. Check whether the generator lacked constraints that operators hold only in private memory.
5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not selected Benchmark Dataset supporting information.
## Fix
- Add representative source Cases or explicit product constraints to the synthesis context.
- Narrow, edit, archive, or delete a Comparison Direction that produces implausible combinations.
- Generate from a specific missing coverage tuple rather than asking for generic diversity.
- Edit a promising candidate when its behavioral boundary is sound and only presentation is weak.
- Reject candidates whose premise is impossible or whose source context cannot be justified.
- Review duplicates before adding any generated Case to a Benchmark.
Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use.
## Prevention
Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Keep Comparison Directions narrow, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot.
{% example-demo title="Example: impossible refund request" %}
Synthesis produces a Case where an anonymous visitor requests an enterprise-contract refund and supplies internal approval metadata in chat. The tuple is valuable—exception request with conflicting authority—but the realization is impossible. The team adds authentication and channel constraints, narrows the Comparison Direction, and regenerates. The realistic candidate still goes through Case review before selection.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Comparison Directions](/docs/assets/comparison-directions)
- [Boundary Cases](/docs/coverage-engineering/boundary-cases)
- [Case Pool](/docs/coverage-engineering/case-pool)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Case Pool surface](/docs/object-model/case-pool)
- [Cases](/docs/object-model/cases)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
{% /related-card-grid %}