# Operating Teammately end to end Generated: 2026-09-13T04:42:15.734Z Source build: local Canonical docs: https://teammately.ai/docs --- id: orientation.end-to-end title: Operating Teammately end to end summary: Operate the current product from project foundations through benchmark coverage, expert contribution, evaluation, and improvement. kind: task product_area: operating_manual status: stable updated: 2026-09-07 canonical: /docs/getting-oriented/operating-teammately-end-to-end --- # Operating Teammately end to end Use this workflow to coordinate the full correctness system while keeping project foundations, benchmark work, expert authority, evaluation evidence, and candidate improvement separate. ## Decision checkpoint | Phase | Owning scope | Exit condition | | --- | --- | --- | | Establish project understanding | Project | Project Agent Brief and Indexed Reference are usable | | Define content and reusable assets | Project | Project Input Schema, Cases, Harnesses, and Coverage Facets are explicit | | Establish benchmark evidence | Benchmark | Dataset snapshot and coverage state are reviewable | | Resolve specialist correctness | Benchmark Contribution and project governance | Attributable artifacts have explicit lifecycle state | | Evaluate candidates | Benchmark version | Exact Runs and case/rubric evidence are available | | Improve behavior | Benchmark version | Goal Contract, candidates, receipts, and frontier are durable | ## Prerequisites - A workspace and project. - An accountable operator, domain expert, and AI engineer or candidate owner. - Source knowledge, examples, and a candidate system appropriate to the intended benchmark. ## Before and after | Before | Operation | After | | --- | --- | --- | | Agents lack a shared project model | Configure Agent Setup | Project understanding is reusable and inspectable | | Examples have inconsistent shape | Save Project Input Schema and prepare Cases | Inputs and materials share a canonical contract | | Coverage and correctness are implicit | Define Coverage Facets and request Contributions | Benchmark intent and specialist standards are explicit | | Candidate claims depend on anecdotes | Run Benchmark Evaluations | Evidence is bound to versions, cases, and rubrics | | Engineering iterations lack chronology | Use Improve | Goals, candidates, receipts, and current frontier stay connected | ## Steps 1. Configure Project Context and Reference Materials in **Agent Setup**. Create or select Comparison Directions and Review Screens under **Assets** when a Contribution needs them. 2. Save Project Input Schema and establish reusable Coverage Facets. 3. Create or import Cases and save candidate Harness versions under Assets. 4. Create or select a benchmark, configure Coverage Management, select Cases in Benchmark Datasets, inspect Representation, and preserve a snapshot. 5. Request focused Expert Contributions for unresolved standards, cases, or coverage. Reconcile contributed artifacts in their owning project or benchmark surfaces. 6. Confirm governed policies and rubrics and the benchmark version that will use them. 7. Run exact saved Harness versions through Benchmark Evaluations. Inspect Dashboard, List, Compare, Arena, and Run detail according to the question. 8. Start an Improvement Session only from evidence that identifies measurable candidate work. 9. Return newly discovered correctness or coverage gaps to Expert Contributions, Correctness Governance, Coverage Management, or Cases. ## Object and state changes This workflow touches project context, reference indexes, input schema, facets, assets, benchmark datasets and snapshots, coverage state, Contributions and contributed artifacts, policies, rubrics, Harness versions, Runs, evaluation results, and Improvement Sessions. Each object remains in its owning scope and retains historical evidence. ## Success criteria - Project and benchmark scope is explicit at every operation. - Agent preparation, expert judgment, and governed artifacts remain distinguishable. - Dataset and candidate versions make evaluation reproducible. - Improvement begins with a measurable goal and pinned evidence. - New learning returns to one responsible upstream artifact. ## Common failure modes - Putting benchmark-specific instructions into permanent Project Context. - Treating connected sources as approved standards. - Selecting generated Cases without case review or schema conformance. - Comparing Runs after multiple evidence boundaries changed. - Treating external-worker activity as observable before an artifact returns. {% example-demo title="Example: full grounding loop" %} A team indexes source repositories, defines source-authority coverage, imports canonical cases, and requests a Contribution to resolve conflicting guidance. The governed rubric enters a benchmark version, two saved Harness versions are compared, and an Improvement Session tests retrieval changes. A missing-source pattern discovered during improvement returns to Coverage Management. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Key objects and relationships](/docs/getting-oriented/key-objects-and-relationships) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Expert Contribution problems](/docs/troubleshooting/expert-contributions) - [Benchmark runs](/docs/troubleshooting/benchmark-runs) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} ## Source confidence Doctrine-backed: the sequence follows the current public capability model and active product topology. Linked pages provide code-backed operation details. --- id: orientation.product-map title: Product map summary: Navigate Teammately across workspace entry points, project foundations, benchmark workspaces, expert contribution UI, and administration. kind: concept product_area: introduction status: stable updated: 2026-09-07 canonical: /docs/getting-oriented/product-map --- # Product map Teammately separates reusable project foundations from benchmark-scoped work. The Main UI uses the selected project and benchmark to route operators to the right scope. Experts receive a focused Expert contribution UI. Admin Console owns organization-level controls, while AI-assisted background work prepares and coordinates bounded tasks. > Surface routing > > Before changing an artifact, identify its scope. Project foundations can affect several benchmarks; dataset selection, Contributions, Evaluations, and Improvement Sessions belong to a selected benchmark or benchmark version. ## Definition The **Main UI** begins at Project Home and groups project-level work into Correctness Governance, Coverage Facets, Assets, Agent Setup, and Project Settings. After a benchmark is selected, its workspace exposes Benchmark Overview, Benchmark Datasets, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improve. The **Expert contribution UI** presents one Contribution and its form, chat, interview, case-review, checkpoint, waiting, and completion states. The expert does not need the full project navigation to supply attributable judgment. The **Admin Console** contains organization administration such as members, groups, roles, domain controls, integrations, and other code-backed administrative surfaces. Public docs keep detailed security, billing, retention, and compliance claims outside the boundary unless separately verified. **AI-assisted background work** can index reference material, prepare contributions, suggest coverage or standards, construct cases, run evaluations, and coordinate candidate exploration. Its outputs retain the authority of the owning artifact and workflow. ## Decision checkpoint | Work | Scope | Surface | | --- | --- | --- | | Project purpose and knowledge | Project | Agent Setup | | Policies and rubrics | Project | Correctness Governance | | Dimensions, Topics, and construction patterns | Project | Coverage Facets | | Reusable Cases and Harnesses | Project | Assets | | Input architecture | Project | Project Settings | | Benchmark-level run fields | Benchmark | Benchmark Evaluations | | Selected Cases, representation, and snapshots | Benchmark | Benchmark Datasets | | Coverage setup, Stories, Case Review, and Foundry | Benchmark | Coverage Management | | Specialist requests and contributed artifacts | Benchmark | Expert Contributions | | Runs, results, Compare, and Arena | Benchmark version | Benchmark Evaluations | | Goal Contracts, candidates, and frontier | Benchmark version | Improve | ## How selection affects navigation Project surfaces require a project. Benchmark surfaces also require a benchmark, and Evaluations or Improve may resolve the current benchmark version. If a destination is unavailable, confirm the current selectors before assuming that the feature or data is missing. Project folders and search help users move across a larger workspace, but they do not change artifact ownership. Search results and creation actions should preserve the selected project or benchmark scope. {% surface-map title="Teammately product surfaces" %} {% /surface-map %} {% example-demo title="Route a new rubric need" %} An evaluation exposes inconsistent handling of expired agreements. The operator uses the benchmark workspace to request an Expert Contribution with the failed cases. The expert works in the focused contribution UI. The resulting rubric is reconciled in Correctness Governance at project scope, then included in a later benchmark version and evaluation. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Task index](/docs/operating-manual/task-index) - [Operating Teammately end to end](/docs/getting-oriented/operating-teammately-end-to-end) - [Product quickstart](/docs/quickstart) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Key objects and relationships](/docs/getting-oriented/key-objects-and-relationships) - [User roles](/docs/getting-oriented/user-roles) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Code-backed: current navigation and active project, benchmark, Contribution, and administration routes establish the scope and labels described here. --- id: product-loop title: The Teammately correctness loop summary: See how coverage, elicitation, case construction, evaluation, and improvement reinforce one another. kind: concept product_area: introduction status: stable updated: 2026-08-22 canonical: /docs/product-loop --- # The Teammately correctness loop The correctness loop is how a team repeatedly turns domain knowledge into stronger AI behavior. It follows the five public capabilities while preserving a trace from every result back to the project context, expert contribution, case, policy, rubric, benchmark version, Harness version, and evaluation setting that made the result meaningful. ## Definition 1. **Design coverage.** Establish Dimensions, Project Topics, and Case Construction Patterns, then decide which combinations the benchmark must represent. 2. **Elicit correctness.** Use focused expert contributions to resolve policies, exceptions, applicability, disagreements, and binary rubric language. 3. **Construct the challenge set.** Create or import canonical cases, attach required materials, generate difficult variants, and curate benchmark dataset membership. 4. **Evaluate behavior.** Run an exact saved Harness Version against an exact Benchmark Version and inspect responses, Case-level Rubric evidence, comparisons, and rankings. 5. **Improve from evidence.** Start an Improvement Session with a bounded Goal Contract, explore candidates, evaluate them through the canonical path, and retain a current frontier. 6. **Return new learning.** Update coverage, correctness, cases, or the candidate according to what the evidence actually showed. ## Decision checkpoint | Evidence says... | Responsible part of the loop | Change first | | --- | --- | --- | | Important behavior has no cases | Coverage Engineering or Weave | Coverage facet, construction pattern, or case set | | Experts cannot apply the standard consistently | Correctness Elicitation | Policy scope, applicability, or rubric wording | | A case cannot be interpreted or executed reliably | Weave and Project Input Schema | Input shape, case material, or world boundary | | One saved candidate fails applicable rubrics | Trialground | Harness candidate or its runtime configuration | | Several candidate branches improve different slices | Coevolve | Goal constraints, next experiment, or retained frontier | | Result movement cannot be explained | Benchmark version and evaluation boundary | Versions, settings, mapping, or run metadata before any product change | ## How expert effort compounds The loop should ask an expert only after agents have prepared the relevant structure and evidence. A Contribution can include selected Cases, source attachments, scoped statements, draft Policies, Rubric questions, or coverage uncertainty. Completed expert work can materialize as an attributable contributed Policy, Rubric, Case, or coverage observation through the owning workflow. That same judgment can guide future case construction, determine which rubrics apply during evaluation, and identify missing correctness during improvement. Reuse across the loop is more valuable than maximizing the number of disconnected review actions. ## How product scope changes through the loop Project foundations are reusable. Project Context, Reference Materials, policies, rubrics, Coverage Facets, Cases, and Harnesses do not belong to only one benchmark. A benchmark workspace selects and versions the relevant subset, manages coverage, coordinates contributions, evaluates candidates, and records improvement. This scope distinction prevents accidental drift. Editing a project-level policy may affect several benchmarks. Changing dataset membership should create a new benchmark evidence boundary. Saving a Harness draft is different from selecting an exact saved Harness version for a Run. ## Before and after | Before | Loop work | After | | --- | --- | --- | | Domain knowledge is distributed across people and files | Agent Setup and Correctness Elicitation organize it | Project context and governed correctness artifacts are inspectable | | Examples are convenient rather than deliberate | Coverage Engineering and Weave shape the challenge set | Dataset representation and missing coverage are explicit | | Candidate behavior is discussed from anecdotes | Trialground runs a versioned evaluation | Case-level rubric evidence and comparisons are available | | Improvement is a sequence of untracked edits | Coevolve starts from pinned evidence | Candidate branches, receipts, chronology, and current frontier remain connected | {% example-demo title="Changing a retrieval harness" %} An evaluation shows failures only when current and superseded documents appear together. The team first confirms that the coverage slice and grounding rubric are valid. An Improvement Session pins those cases and the failing Harness version, then tests source-date filtering and citation-selection candidates. A stronger candidate becomes part of the current frontier only after a canonical evaluation produces the expected rubric evidence. If the work uncovers an unseen source-conflict pattern, that observation returns to Coverage Management. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Product quickstart](/docs/quickstart) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Start an Improvement Session](/docs/improve/start-improvement-session) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Project Input Schema](/docs/project-settings/input-schema) - [Expert Contributions](/docs/expert-contributions) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page explains the approved operating loop. Linked product pages are the authority for exact controls and lifecycle states. --- id: operating.task-index title: Task index summary: Route correctness work to the current project foundation, benchmark workspace, evaluation, or improvement surface. kind: concept product_area: operating_manual status: stable updated: 2026-09-07 canonical: /docs/operating-manual/task-index --- # Task index Use this index when you know the work that must happen but need the current product surface. First decide whether the object is a reusable project foundation or belongs to one benchmark workspace. ## Decision checkpoint | Need | Open | Task | | --- | --- | --- | | Give agents stable project understanding | Agent Setup | [Maintain Project Context](/docs/agent-setup/project-context) | | Connect and verify project knowledge | Agent Setup → Reference Materials | [Use Reference Materials](/docs/agent-setup/reference-materials) | | Define case input and material shape | Project Settings → Input Schema | [Configure Project Input Schema](/docs/project-settings/input-schema) | | Govern policies and rubrics | Correctness Governance | [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics) | | Create or inspect reusable cases | Assets → Cases | [Import and prepare cases](/docs/operating-manual/import-and-prepare-cases) | | Edit a candidate implementation | Assets → Harnesses | [Harnesses](/docs/assets/harnesses) | | Define reusable coverage structure | Coverage Facets | [Coverage Engineering](/docs/coverage-engineering) | | Select benchmark cases and snapshots | Benchmark Datasets | [Benchmark Datasets](/docs/benchmark-datasets) | | Find and close coverage gaps | Coverage Management | [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage) | | Ask a specialist for judgment | Expert Contributions | [Request an Expert Contribution](/docs/expert-contributions/request-contribution) | | Evaluate a saved candidate | Benchmark Evaluations | [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation) | | Diagnose candidate behavior | Dashboard, List, Compare, or Arena | [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) | | Coordinate a justified candidate change | Improve | [Start an Improvement Session](/docs/improve/start-improvement-session) | ## Route by scope Project foundations are reusable across benchmarks. Project Context, Reference Materials, policies, rubrics, Coverage Facets, Cases, Harnesses, and Project Input Schema belong at project scope. Changing one can affect future work in several benchmarks. Benchmark work is deliberately scoped. Dataset selection and snapshots, Coverage Management, Expert Contributions, Benchmark Evaluations, and Improvement Sessions belong to the selected benchmark or benchmark version. Confirm the benchmark selector before making changes or interpreting evidence. ## Route by evidence problem If an evaluation fails, do not assume the Harness is responsible. An unclear Case belongs in Assets or case preparation. Missing behavior belongs in Coverage Management. Ambiguous correctness belongs in an Expert Contribution or Correctness Governance. A changed snapshot, setting, mapping, or metadata value belongs in evaluation diagnosis. Use Improve only when candidate work is justified by pinned evidence. If agents lack source authority, update Reference Materials or Project Context before asking experts or generating more cases. If experts see the wrong fields or interaction, update Review Screen or the scoped Contribution rather than changing benchmark correctness. {% example-demo title="Route a grounding regression" %} A Run regresses on conflicting-source cases. The operator opens List and confirms that the cases, rubric, and settings are valid. Because the candidate selects a superseded document, the work belongs in Improve. If the expert could not determine which source controls, the same evidence would instead route to an Expert Contribution and Correctness Governance. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Product quickstart](/docs/quickstart) - [First correctness loop](/docs/operating-manual/first-correctness-loop) - [Operating Teammately end to end](/docs/getting-oriented/operating-teammately-end-to-end) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Product map](/docs/getting-oriented/product-map) - [Key objects and relationships](/docs/getting-oriented/key-objects-and-relationships) - [Reference library](/docs/reference) {% /related-card-grid %} ## Source confidence Code-backed: the task routing follows current project and benchmark navigation and the active owning routes for each workflow.