The Teammately correctness loop
The correctness loop is how a team repeatedly turns domain knowledge into stronger AI behavior. It follows the five public capabilities while preserving a trace from every result back to the project context, expert contribution, case, policy, rubric, benchmark version, Harness version, and evaluation setting that made the result meaningful.
Definition
- Design coverage. Establish Dimensions, Project Topics, and Case Construction Patterns, then decide which combinations the benchmark must represent.
- Elicit correctness. Use focused expert contributions to resolve policies, exceptions, applicability, disagreements, and binary rubric language.
- Construct the challenge set. Create or import canonical cases, attach required materials, generate difficult variants, and curate benchmark dataset membership.
- Evaluate behavior. Run an exact saved Harness Version against an exact Benchmark Version and inspect responses, Case-level Rubric evidence, comparisons, and rankings.
- Improve from evidence. Start an Improvement Session with a bounded Goal Contract, explore candidates, evaluate them through the canonical path, and retain a current frontier.
- Return new learning. Update coverage, correctness, cases, or the candidate according to what the evidence actually showed.
Decision checkpoint
| Evidence says... | Responsible part of the loop | Change first |
|---|---|---|
| Important behavior has no cases | Coverage Engineering or Weave | Coverage facet, construction pattern, or case set |
| Experts cannot apply the standard consistently | Correctness Elicitation | Policy scope, applicability, or rubric wording |
| A case cannot be interpreted or executed reliably | Weave and Project Input Schema | Input shape, case material, or world boundary |
| One saved candidate fails applicable rubrics | Trialground | Harness candidate or its runtime configuration |
| Several candidate branches improve different slices | Coevolve | Goal constraints, next experiment, or retained frontier |
| Result movement cannot be explained | Benchmark version and evaluation boundary | Versions, settings, mapping, or run metadata before any product change |
How expert effort compounds
The loop should ask an expert only after agents have prepared the relevant structure and evidence. A Contribution can include selected Cases, source attachments, scoped statements, draft Policies, Rubric questions, or coverage uncertainty. Completed expert work can materialize as an attributable contributed Policy, Rubric, Case, or coverage observation through the owning workflow.
That same judgment can guide future case construction, determine which rubrics apply during evaluation, and identify missing correctness during improvement. Reuse across the loop is more valuable than maximizing the number of disconnected review actions.
How product scope changes through the loop
Project foundations are reusable. Project Context, Reference Materials, policies, rubrics, Coverage Facets, Cases, and Harnesses do not belong to only one benchmark. A benchmark workspace selects and versions the relevant subset, manages coverage, coordinates contributions, evaluates candidates, and records improvement.
This scope distinction prevents accidental drift. Editing a project-level policy may affect several benchmarks. Changing dataset membership should create a new benchmark evidence boundary. Saving a Harness draft is different from selecting an exact saved Harness version for a Run.
Before and after
| Before | Loop work | After |
|---|---|---|
| Domain knowledge is distributed across people and files | Agent Setup and Correctness Elicitation organize it | Project context and governed correctness artifacts are inspectable |
| Examples are convenient rather than deliberate | Coverage Engineering and Weave shape the challenge set | Dataset representation and missing coverage are explicit |
| Candidate behavior is discussed from anecdotes | Trialground runs a versioned evaluation | Case-level rubric evidence and comparisons are available |
| Improvement is a sequence of untracked edits | Coevolve starts from pinned evidence | Candidate branches, receipts, chronology, and current frontier remain connected |
Worked example
Changing a retrieval harness
An evaluation shows failures only when current and superseded documents appear together. The team first confirms that the coverage slice and grounding rubric are valid. An Improvement Session pins those cases and the failing Harness version, then tests source-date filtering and citation-selection candidates. A stronger candidate becomes part of the current frontier only after a canonical evaluation produces the expected rubric evidence. If the work uncovers an unseen source-conflict pattern, that observation returns to Coverage Management.
Related workflows
Related reference pages
Source confidence
Doctrine-backed: this page explains the approved operating loop. Linked product pages are the authority for exact controls and lifecycle states.