Teammately Docs
Docs menu

recipe

Building a Correctness Benchmark for a RAG System

Represent retrieval-grounded behavior through cases, context, policies, rubrics, and benchmark evidence.

Building a Correctness Benchmark for a RAG System

Use this playbook when correctness depends on retrieved context, source authority, and whether the answer should cite, abstain, or explain uncertainty.

Entry conditions

Use this when you can preserve the query, retrieved material, candidate response, and source identity for representative RAG behavior. If you have only aggregate retrieval metrics, first collect Case-level evidence; Teammately cannot infer source authority from a score.

Route through Teammately

  1. In Agent Setup, make the Project Agent Brief describe the retrieval architecture and connect the Reference Materials needed to interpret sources.
  2. Configure Project Input Schema fields for the query, retrieved passages, source identifiers, and freshness or authority metadata actually available to the Harness.
  3. Import representative Cases under Assets → Cases. Keep missing-source and conflicting-source Cases instead of filtering them out as bad data.
  4. In Coverage Facets, model the slices that change grounding behavior: authority, freshness, answerability, retrieval completeness, and question type.
  5. Request an Expert Contribution for Cases where the controlling source, required caveat, or abstention boundary is unclear.
  6. Materialize and approve the resulting Policies and binary Rubrics in Correctness Governance.
  7. Use Coverage Management to expose missing combinations, review new Cases, and select the intended set in Benchmark Datasets.
  8. Run saved Harness Versions in Benchmark Evaluations. Read response and Rubric evidence; execution traces are not currently exposed.

Decision gates

  • If the correct source was never retrieved, route the finding to retrieval or coverage work.
  • If the source was present but the response blended, ignored, or contradicted it, route the finding to Harness behavior.
  • If specialists disagree about which source controls, resolve correctness before expanding the Dataset.
  • If an important source condition has too few Cases, hold aggregate interpretation until representation improves.

Worked example

Benefits policy retrieval

An employee asks whether caregiver leave applies to contractors. The Case contains an obsolete handbook page and the current controlling policy, which does not state contractor eligibility. Experts approve a Policy requiring the controlling source and a Rubric that passes only when the answer cites it and withholds the unsupported eligibility claim. Results show one Harness succeeds when both passages are retrieved but still fails when the current policy is absent, separating answer behavior from retrieval coverage.

Evidence to collect

  • Canonical Case input containing the query and material actually available at execution.
  • Source identifiers, authority, and freshness facts that reviewers can verify.
  • Approved grounding, citation, contradiction, and abstention standards.
  • Dataset representation across answerable, conflicting, stale, missing, and multi-source conditions.
  • Saved Harness Version, Benchmark Version, execution settings, Run Metadata, response, and Rubric outcomes.

Related docs

Task / Coverage ManagementSet Up Benchmark CoverageDefine benchmark intent, facet handling, artifact preferences, and evidence requirements before generating coverage work.Reference / Object ModelRepresent conversations in CasesPreserve multi-message context inside canonical Case input without inventing a separate Conversation Case object.Task / Benchmark EvaluationsCompare Harness VersionsCompare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.Task / Benchmark EvaluationsInspect Evaluation ResultsTrace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.Task / Operating ManualImport and prepare casesBring cases into the project, conform them to Project Input Schema, inspect materials, and prepare benchmark selection.

Source confidence

Doctrine-backed: the approved five-capability model establishes the RAG correctness loop. Linked code-backed pages define the current Agent Setup, Case, Coverage Management, Expert Contribution, Dataset, and Evaluation surfaces and their capability fences.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Coverage Engineering Plan Benchmark Coverage Apply project Coverage Facets to one benchmark, inspect representation, and turn important gaps into concrete case or contribution work. Agent Setup Project Context Maintain the Project Agent Brief that gives Teammately agents stable, project-wide understanding. Benchmark Evaluations Inspect Evaluation Results Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.
AI context