# First correctness loop Generated: 2026-09-13T04:38:56.336Z Source build: local Canonical docs: https://teammately.ai/docs --- id: operating.first-correctness-loop title: First correctness loop summary: Complete one traceable path from project context and benchmark coverage to expert judgment, evaluation evidence, and improvement. kind: task product_area: operating_manual status: stable updated: 2026-09-07 canonical: /docs/operating-manual/first-correctness-loop --- # First correctness loop Complete one narrow loop that another operator can reconstruct. Choose one specialist behavior slice and preserve the path from project knowledge through coverage, expert contribution, governed standards, benchmark evidence, and any candidate change. ## Decision checkpoint | State | Next action | Do not continue when... | | --- | --- | --- | | Project intent or sources are implicit | Complete Agent Setup | Agents cannot find the controlling context | | Case shape varies | Configure Project Input Schema | Existing and planned cases do not share a valid contract | | Important behavior is unnamed | Define Coverage Facets and benchmark guidance | The selected cases are merely convenient examples | | Correctness remains tacit | Request a focused Expert Contribution | The expert lacks cases or source evidence | | Cases and standards are ready | Snapshot the dataset and run an evaluation | Candidate, benchmark, mapping, or settings are ambiguous | | Candidate weakness is confirmed | Start an Improvement Session | The target cannot be measured from pinned evidence | ## Prerequisites - One project, one benchmark, and one narrow specialist behavior. - An accountable operator and domain expert. - Representative examples or enough Reference Materials to construct them. - A candidate that can be saved as a Harness version. ## Before and after | Before | Work | After | | --- | --- | --- | | Knowledge is distributed across people and sources | Project Context and Indexed Reference | Agents have inspectable project understanding | | Benchmark examples lack deliberate structure | Coverage Facets, Coverage Management, and dataset selection | The behavior slice and snapshot are explicit | | Judgment is tacit | Expert Contribution and Correctness Governance | Policies and rubrics preserve authority and applicability | | Candidate quality is anecdotal | Benchmark Evaluation | Responses and rubric results bind to exact versions | | Improvement is an informal edit | Improvement Session | Goal, candidate, receipt, and frontier remain connected | ## Steps 1. Write a concise Project Agent Brief and connect the controlling Reference Materials. 2. Configure Project Input Schema for the input architecture and required case materials. 3. Define the relevant Dimensions, Project Topics, and Case Construction Pattern. 4. Add or construct a small case set, inspect its representation, and record any known gap. 5. Request an Expert Contribution with selected cases and a concrete correctness objective. 6. Reconcile the resulting policy, rubric, case, or coverage observation in its owning surface. 7. Select the benchmark dataset cases and create or choose the intended snapshot. 8. Save the candidate Harness version and run a Benchmark Evaluation. 9. Inspect failures at case and rubric level; compare only after confirming evidence boundaries. 10. Start an Improvement Session if candidate work is justified, or return upstream to the specific coverage, correctness, or case artifact that needs change. ## Object and state changes The loop can create or update project context, Reference Materials items and indexed blocks, Project Input Schema, Coverage Facets, Cases, benchmark coverage guidance, Contributions, contributed artifacts, policies, rubrics, dataset selection and snapshots, Harness versions, Runs, evaluation results, and Improvement Sessions. Each transition retains its own authority and scope. ## Success criteria - The selected behavior slice has a named coverage reason. - Expert judgment is attributable and materialized only through an explicit lifecycle. - Case content follows the Project Input Schema. - Evaluation evidence identifies exact candidate and benchmark versions. - The next action names one responsible artifact or candidate boundary. ## Common failure modes - Beginning with a broad benchmark and vague expert request. - Treating Reference Materials as governed standards. - Adding generated cases without a named coverage gap. - Running an editable Harness Draft. - Starting improvement from an aggregate result without pinned measurement evidence. {% example-demo title="Example: one exception slice" %} The first loop targets exception requests with conflicting sources. The project indexes both sources, defines the source-authority facet, asks an expert to establish the controlling rule, creates the corresponding rubric, snapshots ten reviewed cases, evaluates one saved Harness version, and starts improvement from the three exact grounding failures. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Agent Setup](/docs/agent-setup) - [Benchmark Datasets](/docs/benchmark-datasets) - [Expert Contributions](/docs/expert-contributions) - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Improve](/docs/improve) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unclear cases](/docs/troubleshooting/unclear-cases) - [Low expert agreement](/docs/troubleshooting/low-expert-agreement) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} ## Source confidence Doctrine-backed: this workflow applies the current five-capability model and links to code-backed pages for every exact product operation. --- id: quickstart.product title: Product quickstart summary: Configure one project foundation, one benchmark slice, one expert contribution, one evaluation, and one evidence-backed improvement. kind: quickstart product_area: introduction status: stable updated: 2026-09-07 canonical: /docs/quickstart --- # Product quickstart Run one narrow correctness loop. The goal is not a large benchmark; it is a traceable chain from project context and deliberate coverage to expert-grounded standards, a versioned evaluation, and one justified next change. ## When to use it Use this path for a new project or for an existing AI system whose correctness work is scattered across documents, examples, and informal expert feedback. Choose one behavior slice with a clear specialist owner. ## Decision checkpoint | Starting point | First action | Ready to continue when... | | --- | --- | --- | | Agents do not understand the product or domain | Complete Agent Setup | Project Context and controlling Reference Materials are inspectable | | Cases arrive in inconsistent shapes | Configure Project Input Schema | One input architecture and any required case materials are declared | | Important behavior is not represented deliberately | Define Coverage Facets | Dimensions, Project Topics, and Case Construction Patterns name the slice | | Correctness depends on tacit judgment | Request an Expert Contribution | The expert's scope, selected evidence, and required decisions are explicit | | Cases and standards are ready | Create a benchmark snapshot and evaluate a saved Harness version | Exact cases, rubrics, candidate, and settings are bound to the Run | ## Prerequisites - A Teammately project for the specialist AI behavior. - An accountable project operator and at least one domain expert. - A small number of representative examples or enough Reference Materials to construct them. - A candidate system that can be represented by a saved Harness version before evaluation. ## Before and after | Before | Action | After | Stop if... | | --- | --- | --- | --- | | Domain context is implicit | Write the Project Agent Brief and connect Reference Materials | Agents have explicit project understanding | Controlling sources are missing or contradictory without an owner | | Inputs and supporting artifacts vary | Save Project Input Schema | Cases share one canonical content contract | Existing cases cannot satisfy the proposed schema | | Expert knowledge is tacit | Request and complete a focused contribution | Policies, rubrics, cases, or coverage observations can be materialized | The request asks for a label without the evidence needed to explain it | | Candidate behavior is anecdotal | Evaluate a saved Harness version | Results are traceable to cases and applicable rubrics | Dataset snapshot or candidate version is ambiguous | | A weakness is confirmed | Start an Improvement Session from evidence | Candidate work follows a bounded Goal Contract | The requested outcome has no pinned measurement binding | ## Steps 1. Open or create the project and write the Project Agent Brief in **Agent Setup → Project Context**. 2. Add controlling knowledge through **Agent Setup → Reference Materials → Materials**, then inspect the published blocks in **Indexed Reference**. 3. Configure **Project Settings → Input Schema**. Select plain text, chat, or structured input and declare required case materials and accepted artifact families. 4. Create the smallest useful set of Coverage Facets: a Dimension, relevant Project Topics, and a Case Construction Pattern for the chosen behavior slice. 5. Add or construct cases in Assets, then select the intended cases in **Benchmark Datasets**. Confirm Representation and create or choose the appropriate snapshot. 6. In **Expert Contributions**, request one focused contribution. Select the expert, state the objective, attach or select the relevant cases, and include only the contribution components needed to resolve the question. 7. Inspect the completed contribution and materialize accepted policies, rubrics, cases, or coverage observations through their owning surfaces. 8. Save an exact Harness version. In **Benchmark Evaluations**, configure and run it against the selected benchmark version. 9. Inspect Dashboard and List results before using Compare or Arena. Trace important movement to case-level rubric evidence and run metadata. 10. If a candidate change is justified, open **Improve**, start from the relevant evidence, prepare and confirm the Goal Contract, and evaluate candidate work through the canonical Run path. ## Object and state changes This path can create or update Project Context, Reference Materials items and indexed blocks, Project Input Schema, Coverage Facets, Cases, benchmark dataset membership and snapshots, Contributions, contributed artifacts, policies, rubrics, Harness drafts and saved versions, Runs, evaluation results, and Improvement Sessions. Each object keeps its own authority boundary; completing one step does not automatically approve or materialize every downstream artifact. ## Success criteria - Another operator can identify the project context and source material used by agents. - The case set conforms to the Project Input Schema and represents a named coverage slice. - Expert judgment is attributable to a completed Contribution and its accepted artifacts. - The evaluation binds an exact benchmark version to an exact saved Harness version. - Any improvement work starts from pinned evidence and records its Goal Contract, candidate results, and current frontier. ## Common failure modes - Treating Reference Materials as approved policies. - Asking experts broad questions without selected cases or a concrete contribution objective. - Evaluating an unsaved Harness draft or an unclear benchmark snapshot. - Reading only an aggregate score and skipping failed case/rubric pairs. - Starting improvement before the target and measurement evidence are resolved. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Agent Setup](/docs/agent-setup) - [Project Input Schema](/docs/project-settings/input-schema) - [Expert Contributions](/docs/expert-contributions) - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Improve](/docs/improve) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Dataset upload](/docs/troubleshooting/dataset-upload) - [Unclear cases](/docs/troubleshooting/unclear-cases) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} ## Source confidence Doctrine-backed: this quickstart connects the current public story to code-backed product surfaces. Follow the linked pages for exact states and controls. --- id: product-loop title: The Teammately correctness loop summary: See how coverage, elicitation, case construction, evaluation, and improvement reinforce one another. kind: concept product_area: introduction status: stable updated: 2026-08-22 canonical: /docs/product-loop --- # The Teammately correctness loop The correctness loop is how a team repeatedly turns domain knowledge into stronger AI behavior. It follows the five public capabilities while preserving a trace from every result back to the project context, expert contribution, case, policy, rubric, benchmark version, Harness version, and evaluation setting that made the result meaningful. ## Definition 1. **Design coverage.** Establish Dimensions, Project Topics, and Case Construction Patterns, then decide which combinations the benchmark must represent. 2. **Elicit correctness.** Use focused expert contributions to resolve policies, exceptions, applicability, disagreements, and binary rubric language. 3. **Construct the challenge set.** Create or import canonical cases, attach required materials, generate difficult variants, and curate benchmark dataset membership. 4. **Evaluate behavior.** Run an exact saved Harness Version against an exact Benchmark Version and inspect responses, Case-level Rubric evidence, comparisons, and rankings. 5. **Improve from evidence.** Start an Improvement Session with a bounded Goal Contract, explore candidates, evaluate them through the canonical path, and retain a current frontier. 6. **Return new learning.** Update coverage, correctness, cases, or the candidate according to what the evidence actually showed. ## Decision checkpoint | Evidence says... | Responsible part of the loop | Change first | | --- | --- | --- | | Important behavior has no cases | Coverage Engineering or Weave | Coverage facet, construction pattern, or case set | | Experts cannot apply the standard consistently | Correctness Elicitation | Policy scope, applicability, or rubric wording | | A case cannot be interpreted or executed reliably | Weave and Project Input Schema | Input shape, case material, or world boundary | | One saved candidate fails applicable rubrics | Trialground | Harness candidate or its runtime configuration | | Several candidate branches improve different slices | Coevolve | Goal constraints, next experiment, or retained frontier | | Result movement cannot be explained | Benchmark version and evaluation boundary | Versions, settings, mapping, or run metadata before any product change | ## How expert effort compounds The loop should ask an expert only after agents have prepared the relevant structure and evidence. A Contribution can include selected Cases, source attachments, scoped statements, draft Policies, Rubric questions, or coverage uncertainty. Completed expert work can materialize as an attributable contributed Policy, Rubric, Case, or coverage observation through the owning workflow. That same judgment can guide future case construction, determine which rubrics apply during evaluation, and identify missing correctness during improvement. Reuse across the loop is more valuable than maximizing the number of disconnected review actions. ## How product scope changes through the loop Project foundations are reusable. Project Context, Reference Materials, policies, rubrics, Coverage Facets, Cases, and Harnesses do not belong to only one benchmark. A benchmark workspace selects and versions the relevant subset, manages coverage, coordinates contributions, evaluates candidates, and records improvement. This scope distinction prevents accidental drift. Editing a project-level policy may affect several benchmarks. Changing dataset membership should create a new benchmark evidence boundary. Saving a Harness draft is different from selecting an exact saved Harness version for a Run. ## Before and after | Before | Loop work | After | | --- | --- | --- | | Domain knowledge is distributed across people and files | Agent Setup and Correctness Elicitation organize it | Project context and governed correctness artifacts are inspectable | | Examples are convenient rather than deliberate | Coverage Engineering and Weave shape the challenge set | Dataset representation and missing coverage are explicit | | Candidate behavior is discussed from anecdotes | Trialground runs a versioned evaluation | Case-level rubric evidence and comparisons are available | | Improvement is a sequence of untracked edits | Coevolve starts from pinned evidence | Candidate branches, receipts, chronology, and current frontier remain connected | {% example-demo title="Changing a retrieval harness" %} An evaluation shows failures only when current and superseded documents appear together. The team first confirms that the coverage slice and grounding rubric are valid. An Improvement Session pins those cases and the failing Harness version, then tests source-date filtering and citation-selection candidates. A stronger candidate becomes part of the current frontier only after a canonical evaluation produces the expected rubric evidence. If the work uncovers an unseen source-conflict pattern, that observation returns to Coverage Management. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Product quickstart](/docs/quickstart) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Start an Improvement Session](/docs/improve/start-improvement-session) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Project Input Schema](/docs/project-settings/input-schema) - [Expert Contributions](/docs/expert-contributions) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page explains the approved operating loop. Linked product pages are the authority for exact controls and lifecycle states. --- id: operating.task-index title: Task index summary: Route correctness work to the current project foundation, benchmark workspace, evaluation, or improvement surface. kind: concept product_area: operating_manual status: stable updated: 2026-09-07 canonical: /docs/operating-manual/task-index --- # Task index Use this index when you know the work that must happen but need the current product surface. First decide whether the object is a reusable project foundation or belongs to one benchmark workspace. ## Decision checkpoint | Need | Open | Task | | --- | --- | --- | | Give agents stable project understanding | Agent Setup | [Maintain Project Context](/docs/agent-setup/project-context) | | Connect and verify project knowledge | Agent Setup → Reference Materials | [Use Reference Materials](/docs/agent-setup/reference-materials) | | Define case input and material shape | Project Settings → Input Schema | [Configure Project Input Schema](/docs/project-settings/input-schema) | | Govern policies and rubrics | Correctness Governance | [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics) | | Create or inspect reusable cases | Assets → Cases | [Import and prepare cases](/docs/operating-manual/import-and-prepare-cases) | | Edit a candidate implementation | Assets → Harnesses | [Harnesses](/docs/assets/harnesses) | | Define reusable coverage structure | Coverage Facets | [Coverage Engineering](/docs/coverage-engineering) | | Select benchmark cases and snapshots | Benchmark Datasets | [Benchmark Datasets](/docs/benchmark-datasets) | | Find and close coverage gaps | Coverage Management | [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage) | | Ask a specialist for judgment | Expert Contributions | [Request an Expert Contribution](/docs/expert-contributions/request-contribution) | | Evaluate a saved candidate | Benchmark Evaluations | [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation) | | Diagnose candidate behavior | Dashboard, List, Compare, or Arena | [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) | | Coordinate a justified candidate change | Improve | [Start an Improvement Session](/docs/improve/start-improvement-session) | ## Route by scope Project foundations are reusable across benchmarks. Project Context, Reference Materials, policies, rubrics, Coverage Facets, Cases, Harnesses, and Project Input Schema belong at project scope. Changing one can affect future work in several benchmarks. Benchmark work is deliberately scoped. Dataset selection and snapshots, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improvement Sessions belong to the selected benchmark or benchmark version. Confirm the benchmark selector before making changes or interpreting evidence. ## Route by evidence problem If an evaluation fails, do not assume the Harness is responsible. An unclear Case belongs in Assets or case preparation. Missing behavior belongs in Coverage Management. Ambiguous correctness belongs in an Expert Contribution or Correctness Governance. A changed snapshot, setting, mapping, or metadata value belongs in evaluation diagnosis. Use Improve only when candidate work is justified by pinned evidence. If agents lack source authority, update Reference Materials or Project Context before asking experts or generating more cases. If experts see the wrong fields or interaction, update Review Screen or the scoped Contribution rather than changing benchmark correctness. {% example-demo title="Route a grounding regression" %} A Run regresses on conflicting-source cases. The operator opens List and confirms that the cases, rubric, and settings are valid. Because the candidate selects a superseded document, the work belongs in Improve. If the expert could not determine which source controls, the same evidence would instead route to an Expert Contribution and Correctness Governance. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Product quickstart](/docs/quickstart) - [First correctness loop](/docs/operating-manual/first-correctness-loop) - [Operating Teammately end to end](/docs/getting-oriented/operating-teammately-end-to-end) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Key objects and relationships](/docs/getting-oriented/key-objects-and-relationships) - [Reference library](/docs/reference) {% /related-card-grid %} ## Source confidence Code-backed: the task routing follows current project and benchmark navigation and the active owning routes for each workflow.