{"query":"Building a Correctness Benchmark for a RAG System","corpusVersion":"local","generatedAt":"2026-09-14T05:49:57.442Z","results":[{"blockId":"playbooks.rag-benchmark#building-a-correctness-benchmark-for-a-rag-system","pageId":"playbooks.rag-benchmark","title":"Building a Correctness Benchmark for a RAG System","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#building-a-correctness-benchmark-for-a-rag-system","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"building-a-correctness-benchmark-for-a-rag-system","kind":"recipe","productArea":"playbooks","score":9311.526845231961,"reasons":["search_match","exact_match","title_match","display_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"# Building a Correctness Benchmark for a RAG System\n\nUse this playbook when correctness depends on retrieved context, source authority, and whether the answer should cite, abstain, or explain uncertainty."},{"blockId":"playbooks.rag-benchmark#entry-conditions","pageId":"playbooks.rag-benchmark","title":"Entry conditions","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#entry-conditions","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"entry-conditions","kind":"recipe","productArea":"playbooks","score":2760.7996759476523,"reasons":["search_match","exact_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Entry conditions\n\nUse this when you can preserve the query, retrieved material, candidate response, and source identity for representative RAG behavior. If you have only aggregate retrieval metrics, first collect Case-level evidence; Teammately cannot infer source authority from a score."},{"blockId":"playbooks.rag-benchmark#source-confidence","pageId":"playbooks.rag-benchmark","title":"Source confidence","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#source-confidence","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"source-confidence","kind":"recipe","productArea":"playbooks","score":2755.4945352149675,"reasons":["search_match","exact_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Source confidence\n\nDoctrine-backed: the approved five-capability model establishes the RAG correctness loop. Linked code-backed pages define the current Agent Setup, Case, Coverage Management, Expert Contribution, Dataset, and Evaluation surfaces and their capability fences."},{"blockId":"playbooks.rag-benchmark#route-through-teammately","pageId":"playbooks.rag-benchmark","title":"Route through Teammately","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#route-through-teammately","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"route-through-teammately","kind":"recipe","productArea":"playbooks","score":2690.259820842782,"reasons":["search_match","exact_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Route through Teammately\n\n1. In **Agent Setup**, make the Project Agent Brief describe the retrieval architecture and connect the Reference Materials needed to interpret sources.\n2. Configure Project Input Schema fields for the query, retrieved passages, source identifiers, and freshness or authority metadata actually available to the Harness.\n3. Import representative Cases under **Assets → Cases**. Keep missing-source and conflicting-source Cases instead of filtering them out as bad data.\n4. In **Coverage Facets**, model the slices that change grounding behavior: authority, freshness, answerability, retrieval completeness, and question type.\n5. Request an **Expert Contribution** for Cases where the controlling source, required caveat, or abstention boundary is unclear.\n6. Materialize and approve the resulting Policies and binary Rubrics in **Correctness Governance**.\n7. Use **Coverage Management** to expose missing combinations, review new Cases, and select the intended set in **Benchmark Datasets**.\n8. Run saved Harness Versions in **Benchmark Evaluations**. Read response and Rubric evidence; execution traces are not currently exposed."},{"blockId":"playbooks.rag-benchmark#decision-gates","pageId":"playbooks.rag-benchmark","title":"Decision gates","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#decision-gates","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"decision-gates","kind":"recipe","productArea":"playbooks","score":2680.934656725727,"reasons":["search_match","exact_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Decision gates\n\n- If the correct source was never retrieved, route the finding to retrieval or coverage work.\n- If the source was present but the response blended, ignored, or contradicted it, route the finding to Harness behavior.\n- If specialists disagree about which source controls, resolve correctness before expanding the Dataset.\n- If an important source condition has too few Cases, hold aggregate interpretation until representation improves.\n\n{% example-demo title=\"Benefits policy retrieval\" %}\nAn employee asks whether caregiver leave applies to contractors. The Case contains an obsolete handbook page and the current controlling policy, which does not state contractor eligibility. Experts approve a Policy requiring the controlling source and a Rubric that passes only when the answer cites it and withholds the unsupported eligibility claim. Results show one Harness succeeds when both passages are retrieved but still fails when the current policy is absent, separating answer behavior from retrieval coverage.\n{% /example-demo %}"},{"blockId":"playbooks.rag-benchmark#related-docs","pageId":"playbooks.rag-benchmark","title":"Related docs","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#related-docs","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"related-docs","kind":"recipe","productArea":"playbooks","score":2674.2832897701414,"reasons":["search_match","exact_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Related docs\n\n{% related-card-grid title=\"Related docs\" %}\n- [Configure Coverage Management](/docs/coverage-management/get-started)\n- [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases)\n- [Compare Harness Versions](/docs/benchmark-evaluations/compare)\n- [Read run results](/docs/benchmark-evaluations/inspect-results)\n- [Run a benchmark](/docs/benchmark-evaluations/run-evaluation)\n- [Importing cases](/docs/operating-manual/import-and-prepare-cases)\n{% /related-card-grid %}"},{"blockId":"playbooks.rag-benchmark#evidence-to-collect","pageId":"playbooks.rag-benchmark","title":"Evidence to collect","pageTitle":"Building a Correctness Benchmark for a RAG System","url":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","humanUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag#evidence-to-collect","markdownUrl":"https://teammately.ai/docs/playbooks/building-correctness-benchmark-rag.md","sectionId":"evidence-to-collect","kind":"recipe","productArea":"playbooks","score":2650.5832216951767,"reasons":["search_match","exact_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Evidence to collect\n\n- Canonical Case input containing the query and material actually available at execution.\n- Source identifiers, authority, and freshness facts that reviewers can verify.\n- Approved grounding, citation, contradiction, and abstention standards.\n- Dataset representation across answerable, conflicting, stale, missing, and multi-source conditions.\n- Saved Harness Version, Benchmark Version, execution settings, Run Metadata, response, and Rubric outcomes."},{"blockId":"playbooks.overview#choose-a-playbook","pageId":"playbooks.overview","title":"Choose a playbook","pageTitle":"Enterprise Playbooks","url":"https://teammately.ai/docs/playbooks.md","humanUrl":"https://teammately.ai/docs/playbooks#choose-a-playbook","markdownUrl":"https://teammately.ai/docs/playbooks.md","sectionId":"choose-a-playbook","kind":"concept","productArea":"playbooks","score":1474.3006892015621,"reasons":["search_match","exact_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Choose a playbook\n\n| Starting problem | Playbook |\n| --- | --- |\n| Retrieved sources, citation, abstention, or answer grounding | [RAG correctness benchmark](/docs/playbooks/building-correctness-benchmark-rag) |\n| Search intent, source authority, document conflicts, or freshness | [Enterprise search](/docs/playbooks/enterprise-search) |\n| Refunds, commitments, account context, or escalation | [Customer support AI](/docs/playbooks/customer-support-ai) |\n| Many rules, exceptions, or controlled source hierarchies | [Policy-heavy AI systems](/docs/playbooks/policy-heavy-ai-systems) |\n| A bounded question requires accountable specialist judgment | [Run Expert Contributions](/docs/playbooks/running-expert-contributions-enterprise-assistant) |\n| Contribution evidence needs to become reusable standards | [Turn judgment into Policies and Rubrics](/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics) |\n| Qualified experts disagree | [Handle conflicting opinions](/docs/playbooks/handling-conflicting-expert-opinions) |\n| Important behavior may be absent from the selected Dataset | [Find coverage gaps](/docs/playbooks/finding-coverage-gaps-before-review) |\n| New evidence or a changed rule makes the current boundary stale | [Refresh a Benchmark](/docs/playbooks/refreshing-a-benchmark-from-new-signals) |\n| CI or another evaluation system already owns execution facts | [Use existing evaluation infrastructure](/docs/playbooks/using-teammately-alongside-existing-evaluation-infrastructure) |"}]}