# The Teammately correctness loop Generated: 2026-09-13T04:33:47.146Z Source build: local Canonical docs: https://teammately.ai/docs --- id: product-loop title: The Teammately correctness loop summary: See how coverage, elicitation, case construction, evaluation, and improvement reinforce one another. kind: concept product_area: introduction status: stable updated: 2026-08-22 canonical: /docs/product-loop --- # The Teammately correctness loop The correctness loop is how a team repeatedly turns domain knowledge into stronger AI behavior. It follows the five public capabilities while preserving a trace from every result back to the project context, expert contribution, case, policy, rubric, benchmark version, Harness version, and evaluation setting that made the result meaningful. ## Definition 1. **Design coverage.** Establish Dimensions, Project Topics, and Case Construction Patterns, then decide which combinations the benchmark must represent. 2. **Elicit correctness.** Use focused expert contributions to resolve policies, exceptions, applicability, disagreements, and binary rubric language. 3. **Construct the challenge set.** Create or import canonical cases, attach required materials, generate difficult variants, and curate benchmark dataset membership. 4. **Evaluate behavior.** Run an exact saved Harness Version against an exact Benchmark Version and inspect responses, Case-level Rubric evidence, comparisons, and rankings. 5. **Improve from evidence.** Start an Improvement Session with a bounded Goal Contract, explore candidates, evaluate them through the canonical path, and retain a current frontier. 6. **Return new learning.** Update coverage, correctness, cases, or the candidate according to what the evidence actually showed. ## Decision checkpoint | Evidence says... | Responsible part of the loop | Change first | | --- | --- | --- | | Important behavior has no cases | Coverage Engineering or Weave | Coverage facet, construction pattern, or case set | | Experts cannot apply the standard consistently | Correctness Elicitation | Policy scope, applicability, or rubric wording | | A case cannot be interpreted or executed reliably | Weave and Project Input Schema | Input shape, case material, or world boundary | | One saved candidate fails applicable rubrics | Trialground | Harness candidate or its runtime configuration | | Several candidate branches improve different slices | Coevolve | Goal constraints, next experiment, or retained frontier | | Result movement cannot be explained | Benchmark version and evaluation boundary | Versions, settings, mapping, or run metadata before any product change | ## How expert effort compounds The loop should ask an expert only after agents have prepared the relevant structure and evidence. A Contribution can include selected Cases, source attachments, scoped statements, draft Policies, Rubric questions, or coverage uncertainty. Completed expert work can materialize as an attributable contributed Policy, Rubric, Case, or coverage observation through the owning workflow. That same judgment can guide future case construction, determine which rubrics apply during evaluation, and identify missing correctness during improvement. Reuse across the loop is more valuable than maximizing the number of disconnected review actions. ## How product scope changes through the loop Project foundations are reusable. Project Context, Reference Materials, policies, rubrics, Coverage Facets, Cases, and Harnesses do not belong to only one benchmark. A benchmark workspace selects and versions the relevant subset, manages coverage, coordinates contributions, evaluates candidates, and records improvement. This scope distinction prevents accidental drift. Editing a project-level policy may affect several benchmarks. Changing dataset membership should create a new benchmark evidence boundary. Saving a Harness draft is different from selecting an exact saved Harness version for a Run. ## Before and after | Before | Loop work | After | | --- | --- | --- | | Domain knowledge is distributed across people and files | Agent Setup and Correctness Elicitation organize it | Project context and governed correctness artifacts are inspectable | | Examples are convenient rather than deliberate | Coverage Engineering and Weave shape the challenge set | Dataset representation and missing coverage are explicit | | Candidate behavior is discussed from anecdotes | Trialground runs a versioned evaluation | Case-level rubric evidence and comparisons are available | | Improvement is a sequence of untracked edits | Coevolve starts from pinned evidence | Candidate branches, receipts, chronology, and current frontier remain connected | {% example-demo title="Changing a retrieval harness" %} An evaluation shows failures only when current and superseded documents appear together. The team first confirms that the coverage slice and grounding rubric are valid. An Improvement Session pins those cases and the failing Harness version, then tests source-date filtering and citation-selection candidates. A stronger candidate becomes part of the current frontier only after a canonical evaluation produces the expected rubric evidence. If the work uncovers an unseen source-conflict pattern, that observation returns to Coverage Management. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Product quickstart](/docs/quickstart) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Start an Improvement Session](/docs/improve/start-improvement-session) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Project Input Schema](/docs/project-settings/input-schema) - [Expert Contributions](/docs/expert-contributions) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page explains the approved operating loop. Linked product pages are the authority for exact controls and lifecycle states. --- id: intro.correctness-infrastructure title: What is correctness infrastructure? summary: Learn how five connected capabilities turn specialist judgment into cases, executable standards, evaluation evidence, and improvement. kind: concept product_area: introduction status: stable updated: 2026-08-22 canonical: /docs/introduction/correctness-infrastructure --- # What is correctness infrastructure? Correctness infrastructure is the operating layer that lets a team specify, test, and improve the behavior of specialist AI. It connects the behavior space that matters, the expert judgment that defines acceptable behavior, the cases that challenge a system, the evidence produced by repeatable evaluation, and the engineering work that follows. {% visual-hero src="/docs-assets/assets/correctness-infrastructure-workbench.png" alt="Workbench connecting coverage design, expert judgment, cases, evaluation evidence, and improvement." %} The visual is a category anchor. The selectable capability names and current product mappings below are authoritative. {% /visual-hero %} ## Definition Teammately expresses correctness infrastructure through five capabilities: 1. **Coverage Engineering** designs what the benchmark must represent. 2. **Correctness Elicitation** captures policies, exceptions, applicability, and expert rubrics. 3. **Weave** constructs cases, variants, multimodal materials, and supported worlds from the coverage design. 4. **Trialground** runs harnesses and weights against benchmark cases and records rubric evidence. 5. **Coevolve** explores stronger candidate branches and missing correctness or coverage. The sequence is connected rather than strictly linear. An evaluation failure may expose a missing rubric. An expert contribution may reveal a new coverage facet. An Improvement Session may find that the benchmark needs a harder case before a harness change can be trusted. ## Decision checkpoint | Question | Inspect first | Expected durable result | | --- | --- | --- | | Does the benchmark represent the important behavior space? | Coverage Engineering | Dimensions, Project Topics, Case Construction Patterns, and benchmark coverage state | | What should count as correct for this situation? | Correctness Elicitation | Governed policies, applicability, and binary rubrics | | Do we have cases that exercise the difficult combinations? | Weave | Reviewable cases, variants, materials, and dataset membership candidates | | How did this candidate behave against the benchmark? | Trialground | Runs, responses, Rubric results, comparisons, and rankings | | What change is justified by the evidence? | Coevolve | A bounded Improvement Session with candidates and evaluation receipts | > Keep the category centered > > Evaluation execution is one capability inside correctness infrastructure. The category also includes the expert-derived specification, deliberate coverage, case construction, and evidence-backed improvement that make evaluation meaningful. ## From expert effort to reusable infrastructure Expert time is most valuable when it resolves ambiguity that agents and engineers cannot settle from existing evidence. Teammately therefore prepares a structured contribution: the relevant cases, reference materials, candidate interpretations, possible policies, rubric questions, and unresolved conflicts. Once an expert responds, the contribution can affect more than the immediate task. It can refine the coverage map, materialize a policy or rubric, qualify a case, or identify the next evaluation. This creates a higher return on expert effort. The product does not ask specialists to repeatedly label disconnected outputs; it preserves why a judgment was made and where that judgment applies. ## Product realization The project foundation holds reusable context and assets. Correctness Governance owns policies and rubrics. Coverage Facets owns Dimensions, Project Topics, and Case Construction Patterns. Assets owns cases, harnesses, and other project-level resources, including reusable Comparison Directions and Review Screens. Agent Setup gives agents reusable project understanding through Project Context and Reference Materials. Expert Contributions configure scoped agent behavior and select the presentation and variation assets needed for the work. A benchmark workspace then creates a scoped operational loop. Benchmark Datasets defines the case set and snapshots. Coverage Management exposes represented and missing behavior. Expert Contributions requests governed human work. Benchmark Evaluations produces versioned evidence. Improve starts from that evidence and coordinates candidate exploration. {% layer-map title="Five capabilities of correctness infrastructure" %} {% /layer-map %} {% example-demo title="Policy exception handling" %} A support assistant must answer exception requests. Coverage Engineering identifies exception type, customer state, and source freshness as important facets. Correctness Elicitation records the expert rule for when an exception can be stated or must be escalated. Weave prepares ordinary, conflicting-source, and boundary cases. Trialground shows that a candidate handles ordinary cases but invents approval on ambiguous ones. Coevolve tests a harness change and keeps the resulting candidate tied to the exact failed cases and rubric evidence. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [The correctness lifecycle](/docs/introduction/correctness-lifecycle) - [The correctness loop](/docs/product-loop) - [Product quickstart](/docs/quickstart) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Key objects and relationships](/docs/getting-oriented/key-objects-and-relationships) - [Object model](/docs/object-model) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines the current public category and capability model. Product-surface pages provide code-backed operational detail. --- id: concepts.correctness-elicitation title: Correctness Elicitation summary: Turn tacit specialist judgment into attributable contributions, governed policies, applicability conditions, and binary rubrics. kind: concept product_area: correctness_elicitation status: stable updated: 2026-09-07 canonical: /docs/concepts/correctness-elicitation --- # Correctness Elicitation Correctness Elicitation is the capability for turning specialist judgment into explicit, reusable correctness specifications. It handles the parts of AI behavior that cannot be settled by a generic score: domain preferences, exceptions, conflicts between sources, applicability boundaries, unacceptable failure modes, and the evidence an expert needs before making a decision. ## Definition Elicitation begins before the expert opens a task. Teammately agents can organize relevant Reference Materials, cases, candidate responses, possible policies, rubric questions, and unresolved conflicts into a focused Contribution. The expert then works through forms, chat, interviews, case review, or checkpoints according to the requested components. The contribution remains attributable. Accepted learning can be materialized into policies, rubrics, cases, or coverage observations, while drafts and unresolved statements keep their own state. Correctness Governance owns the resulting policies and rubrics; Expert Contributions owns the benchmark-scoped human work that produced them. ## Decision checkpoint | Need | Elicitation method | Durable destination | | --- | --- | --- | | Confirm a known rule across several cases | Focused form or case review | Policy, applicability, or rubric contribution | | Discover reasoning that is hard to pre-structure | Chat or interview | Attributable transcript, checkpoints, and contributed artifacts | | Resolve disagreement or ambiguity | Targeted comparison and checkpoint | Explicit unresolved or accepted statement | | Identify missing benchmark behavior | Cases plus coverage questions | Contributed case or coverage observation | | Configure reviewer presentation | Assets → Review Screens | Reusable Review Screen | | Set a contribution's agent behavior | Expert Contribution | Contribution behavior and selected components | ## Why preparation matters An expert should not have to reconstruct the project, search for the controlling source, or infer why a case was selected. Project Context gives the agent the stable brief. Reference Materials provide indexed project knowledge. The Contribution selects the benchmark evidence, states the objective, and configures agent behavior. Review Screen controls presentation. This separation preserves authority. Reference material can inform an answer without becoming a policy. Agent-authored draft wording can focus the expert without becoming approved. A completed expert task can contribute evidence without automatically changing every project artifact. ## Relationship to the product Correctness Elicitation is broader than any single screen. Use **Expert Contributions** to request and track benchmark-scoped expert work. Use **Correctness Governance** to inspect and maintain policies and rubrics after they are materialized. Use **Agent Setup** to configure what agents understand, **Assets → Review Screens** for reusable expert-facing presentation, and the Contribution itself for scoped behavior and components. The capability also returns learning to Coverage Engineering and Weave. An expert may identify a missing behavior combination, contribute a new case, or explain that existing material is insufficient. Those outputs should update the owning coverage or case artifacts instead of being flattened into a general review note. {% example-demo title="Exception handling" %} An agent prepares three cases where a procurement rule might allow an exception, attaches the controlling policy material, and asks a specialist to distinguish approved exceptions from escalation-only situations. The specialist completes a case review and confirms a checkpoint. The accepted contribution materializes a scoped policy and binary rubric, while one unresolved source conflict becomes a coverage observation for follow-up. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) - [Manage policies and rubrics](/docs/correctness-governance) - [Configure Agent Setup](/docs/agent-setup) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) - [Policies](/docs/object-model/policies) - [Rubrics](/docs/object-model/rubrics) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines the public capability. The linked product pages are code-backed and define the active contribution, governance, and agent-configuration surfaces. --- id: concepts.weave title: Weave summary: Construct deliberate challenge sets from coverage structure, canonical cases, variants, multimodal materials, and supported worlds. kind: concept product_area: weave status: stable updated: 2026-09-07 canonical: /docs/concepts/weave --- # Weave Weave is the case-construction capability inside Teammately correctness infrastructure. It turns structured coverage intent into the cases, response variants, supporting materials, and supported environments needed to challenge a specialist AI system. Its goal is deliberate representation, not undirected synthetic volume. ## Definition Coverage Engineering supplies the structure: Dimensions, Project Topics, Case Construction Patterns, coverage tuples, and known gaps. Weave uses that structure to prepare candidate cases that exercise meaningful combinations and boundaries. Cases follow the project's Input Schema, so their primary input and required materials have a consistent, inspectable shape. A case can include text, chat messages, structured data, images, documents, tabular files, presentations, source text, or audio when the Project Input Schema accepts those artifacts. Static support belongs to case materials. An executable or queryable environment is a separate world boundary and should only be used where the product supplies an inspectable world reference and runtime. ## Decision checkpoint | Construction need | Use | Verify before benchmark membership | | --- | --- | --- | | Add a known real example | Assets → Cases or dataset import | Input and materials conform to Project Input Schema | | Cover a missing facet combination | Coverage Management and Case Foundry | The case targets a named tuple or gap | | Generate a meaningful output variation | Case construction plus Comparison Directions | The idea guides variation without pretending the output is already reviewed | | Include supporting files | Canonical case materials | Artifact family and extension are accepted by the project | | Exercise changing state or tools | Supported World | World identity, capabilities, visibility, and lifecycle are explicit | ## Deliberate construction Case count is a weak target. A useful challenge set explains why every important slice exists and which behavior boundary it exercises. Case Construction Patterns provide reusable ways to instantiate a behavior problem. Coverage Management shows where cases already exist and where the benchmark remains thin. Case Review lets operators inspect generated materials before those cases become trusted dataset members. Comparison Directions are reusable Assets selected by comparative Contributions. They guide how candidate responses should vary—for example, changing evidence quality or reasoning approach—without serving as a coverage taxonomy or an automatically accepted collection of variants. ## Canonical case boundary `content.input` is the primary case input. Optional `content.case_materials` contains declared supporting values or artifacts. `record_content.case_view` is a rendered projection for users and execution adapters; it is not a second source of truth. Static runtime delivery uses case-material references. Executable worlds use a world instance reference. This distinction keeps the same case understandable to case authors, experts, harness code, and evaluation services. It also lets the system reject material that is missing, undeclared, or incompatible before a Run produces misleading evidence. {% example-demo title="Conflicting-source case" %} Coverage Management identifies a thin tuple combining policy exception, superseded source, and high customer impact. A Case Construction Pattern describes how to pair a current rule with a plausible older document. Weave creates a chat input and attaches both documents as case materials. Case Review confirms that the conflict is visible and realistic before the case enters the benchmark dataset snapshot. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Configure Project Input Schema](/docs/project-settings/input-schema) - [Manage benchmark coverage](/docs/coverage-management) - [Work with Benchmark Datasets](/docs/benchmark-datasets) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Assets and cases](/docs/assets) - [Cases](/docs/object-model/cases) - [Comparison Directions](/docs/assets/comparison-directions) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines Weave as the public construction capability. It deliberately limits world claims to the verified runtime boundary and delegates current controls to linked code-backed pages. --- id: concepts.trialground title: Trialground summary: Evaluate exact Harness and Benchmark Versions in a managed proving ground with inspectable responses and Rubric evidence. kind: concept product_area: trialground status: stable updated: 2026-09-07 canonical: /docs/concepts/trialground --- # Trialground Trialground is the evaluation capability inside Teammately correctness infrastructure. It gives AI engineers a managed proving ground for running exact candidate implementations against exact Benchmark Versions while preserving the response, Rubric outcomes, settings, and metadata needed to explain the result. ## Definition A candidate is represented by a saved Harness version rather than an informal description of a prompt or code state. A benchmark version provides the cases and governed correctness boundary. Benchmark Evaluations binds those inputs into Runs, computes rubric-level evidence, and exposes results through Dashboard, List, Compare, Arena, and run-detail views. Trialground names the capability; **Harnesses** and **Benchmark Evaluations** are the current UI surfaces. The distinction matters when giving instructions: users edit and save a Harness under Assets, then select its saved version for a benchmark evaluation. ## Decision checkpoint | Question | Inspect | Required boundary | | --- | --- | --- | | What code or prompt was evaluated? | Harness and saved version | Draft editor state is not the candidate identity | | What behavior space was tested? | Benchmark version and dataset snapshot | Dataset membership is fixed for interpretation | | Which standards judged each case? | Applicable policies and rubrics | Suggested or unrelated standards are excluded | | Why did a candidate pass or fail? | Case response and applicable Rubric outcomes | Aggregate movement is traced to Case-level evidence | | Can two candidates be compared? | Compare or Arena | Both candidates expose compatible benchmark and settings boundaries | ## Managed evaluation Harness Assets provide a multi-file editing surface, Draft and Saved states, coding-agent assistance, secret references, and benchmark activation. Evaluation setup selects the relevant Harness version and preserves run metadata. Managed execution can scale case work while keeping every result attached to the benchmark and candidate identities. The current Benchmark Evaluations API does not expose execution trajectories, spans, private reasoning, or tool traces. The visible **Traces / Spans** segment is therefore a capability fence, not inspectable Run evidence. Improve's narrated trajectories are separate session chronology and must not be described as evaluation execution traces. Rubric outcomes remain the governed evidence within the evaluation contract. ## Reading evidence Start with Dashboard to understand overall progress and distribution, then use List for Run and Case-level results. Compare is a symmetric matrix whose columns are saved Harness Versions and whose rows can group Cases, all results, Policies, Rubrics, Dimensions, Topics, Groups, or Patterns. Arena computes pair evidence from admitted results; it is not a subjective preference interview. Run detail is the place to resolve execution state, output mapping, response evidence, and failures. When a result is surprising, check versions and configuration before changing the candidate. A different benchmark snapshot, Harness version, output mapping, evaluation setting, or run-metadata value can explain movement that otherwise looks like a behavior change. {% example-demo title="Retrieval candidate comparison" %} Two saved Harness versions use different source-selection strategies. Both run against the same benchmark version containing current-source, stale-source, and missing-source cases. Compare shows that one candidate improves current-source grounding but regresses when no controlling source exists. The team opens the failed cases and rubric results before deciding whether the candidate belongs in the next Improvement Session. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Start an Improvement Session](/docs/improve/start-improvement-session) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Harnesses](/docs/assets/harnesses) - [Benchmarks](/docs/object-model/benchmarks) - [Run metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines Trialground as the public capability. Harness and Benchmark Evaluation pages provide code-backed behavior and state details. --- id: concepts.coevolve title: Coevolve summary: Explore multiple evidence-backed candidate directions while keeping goals, benchmark receipts, trajectories, and the current frontier connected. kind: concept product_area: coevolve status: stable updated: 2026-08-22 canonical: /docs/concepts/coevolve --- # Coevolve Coevolve is the improvement capability inside Teammately correctness infrastructure. It lets agents and engineers pursue multiple candidate directions from benchmark evidence, evaluate those candidates through the same canonical path, and continue from stronger branches without losing the goal, chronology, or evidence behind the current frontier. ## Definition Coevolve is realized through **Improve** and durable **Improvement Sessions**. A session starts from selected benchmark evidence and a target Harness. A Goal Contract turns a free-form objective into pinned target identities, measurement bindings, constraints, unresolved items, and permitted intervention boundaries. Once confirmed, that contract governs future work without rewriting earlier evidence. Sessions can use Work or Evolve behavior. Work supports bounded implementation and review. Evolve supports parallel candidate exploration across epochs and retains a frontier based on current evaluation evidence. Candidate Harness versions become meaningful only when canonical Runs return observable results. ## Decision checkpoint | Situation | Improvement action | Evidence requirement | | --- | --- | --- | | One known candidate change needs implementation | Start a Work session | Confirmed Goal Contract and pinned target evidence | | Several hypotheses should compete | Start an Evolve session | Explicit authorization, measurement bindings, and comparable evaluation path | | An external coding worker will implement | Prepare a scoped worker package | Returned Harness version or evaluation request before claiming observable progress | | A candidate looks stronger | Inspect the current frontier | Canonical evaluation receipts support the retained position | | Exploration exposes missing correctness or coverage | Return the observation upstream | Identify the policy, rubric, case, or coverage artifact that must change | ## Evidence-backed branching Candidate exploration is not a sequence of undocumented edits. Each proposal should state the hypothesis and its relationship to the Goal Contract. Evaluation receipts bind candidate identity to benchmark identity and result. Narrated trajectories can explain the work performed, while chronology records durable transitions, agent activity, pauses, resumptions, and terminal state. The current frontier is not simply the newest candidate. It represents the candidates retained by the session's evidence and goal constraints. A candidate can improve one slice and regress another; the frontier and comparison views keep that tradeoff visible. ## Bridge between experts and engineers Improvement can reveal that the candidate is not the only incomplete part of the system. A missing coverage tuple, unclear rubric, contradictory policy, or insufficient case material should become an upstream contribution opportunity. This is how Coevolve connects coding agents to domain experts: engineering work is guided by benchmark evidence, and newly discovered correctness questions return to focused expert work. External workers remain bounded. Teammately may prepare a package for Codex, Claude Code, or another worker, but it records only the work returned through the defined contract. It does not infer private activity or fabricate a working state. {% example-demo title="Parallel grounding hypotheses" %} An Evolve session starts from failures involving conflicting policy documents. One candidate changes retrieval filtering, another changes source ranking, and a third changes answer construction. Each saved Harness version is evaluated against the pinned benchmark. The frontier retains the candidates supported by grounding and uncertainty rubrics, while a newly observed source-authority ambiguity becomes an Expert Contribution opportunity. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Improve](/docs/improve) - [Harnesses](/docs/assets/harnesses) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines Coevolve as the public capability. The Improve pages provide code-backed session, contract, candidate, and frontier behavior.