Teammately Docs
Docs menu

task

Set Up Benchmark Coverage

Define benchmark intent, facet handling, artifact preferences, and evidence requirements before generating coverage work.

Set Up Benchmark Coverage

Prerequisites

  • Project Coverage Facets and Input Schema are available.
  • The benchmark intent and evidence risk can be stated concretely.

Get Started records the coverage contract that drives Coverage Stories and Case Foundry. Complete it before treating generated coverage work as aligned to the benchmark.

Define the intent and facet treatment

Describe the benchmark intent and concrete requirements. For each Dimension ontology value, choose the benchmark role required by the setup. Configure Topic Group handling and Case Construction Pattern behavior rather than assuming every active project facet must be represented equally.

Dimension roles, Topic Group handling, and Pattern modes are different controls. A required Dimension value constrains represented behavior. A Topic Group can require every Topic, require group-level coverage, provide guidance, or be excluded. A Pattern can be left to the system, preferred, or avoided. Preserve those distinctions when explaining the resulting coverage plan.

Define artifact and evidence expectations

For image, document, tabular, presentation, source-text, and audio artifacts, choose mandatory, optional, or never. Set portfolio limits and accepted formats so construction does not create unsupported or gratuitous material.

The evidence profile can specify actor, workflow, grounding, evidence carriers, difficulty, transformation, and data-handling expectations. These fields make a behavior testable. They are not decorative prose: Case Review uses them to judge whether a prepared candidate proves the intended situation.

Save and check readiness

Setup moves through draft, ready, generated, changed_since_generation, and archived states. Resolve the readiness guidance before generation. If the setup changes after stories or cases were generated, treat the previous work as based on an older input rather than silently presenting it as current.

A coverage guideline can apply to foundry_only or overall_coverage. Overall coverage can require provenance backfill for existing Cases. The coverage compiler can preview reconciled revisions, but an operator confirms the durable update.

Object and state changes

Saving creates or revises benchmark-scoped coverage setup and its readiness status. Generation records which setup revision it used. Archiving stops the setup from acting as the current contract without erasing history.

Success criteria

  • Intent, requirements, facet treatment, artifacts, and evidence profile agree.
  • Readiness is explicit and downstream generation can identify the exact setup.
  • Overall-coverage provenance needs are handled deliberately.

Common failure modes

  • Requiring every active facet without regard to benchmark intent.
  • Marking unsupported artifacts mandatory.
  • Editing setup after generation and ignoring the stale result.

Related reference pages

Concept / Coverage ManagementCoverage ManagementManage benchmark coverage from setup through representation, Coverage Stories, case review, Case Foundry, and contribution requests.Reference / Project SettingsProject Input SchemaDefine the canonical input architecture, case-material fields, and accepted artifact formats for project cases.Reference / Coverage EngineeringDimensions and OntologyDefine reusable behavior axes and their allowed values, then inspect how cases and benchmarks cover them.

Related troubleshooting pages

Troubleshooting / Recovery LibraryUnbalanced CoverageFix benchmarks that overrepresent easy or common cases while missing important behavior.Troubleshooting / Recovery LibrarySynthetic Cases That Feel UnrealisticImprove generated candidates whose surface details or behavior assumptions do not represent plausible target-system use.

Source confidence

Code-backed: the active setup page, types, readiness rules, and save states define the benchmark-scoped coverage contract and its lifecycle.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Management Coverage Management Manage benchmark coverage from setup through representation, Coverage Stories, case review, Case Foundry, and contribution requests. Coverage Management Coverage Stories Organize benchmark coverage intent into governed Stories and testable facet tuples. Coverage Engineering Dimensions and Ontology Define reusable behavior axes and their allowed values, then inspect how cases and benchmarks cover them. Coverage Engineering Project Topics Maintain source-grounded subject areas, editable groups, and Atlas relationships used to organize project and benchmark coverage.
AI context