{"query":"Turning Expert Judgment into Policies and Rubrics","corpusVersion":"local","generatedAt":"2026-09-13T04:42:29.147Z","results":[{"blockId":"playbooks.expert-judgment-to-standards#turning-expert-judgment-into-policies-and-rubrics","pageId":"playbooks.expert-judgment-to-standards","title":"Turning Expert Judgment into Policies and Rubrics","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#turning-expert-judgment-into-policies-and-rubrics","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"turning-expert-judgment-into-policies-and-rubrics","kind":"recipe","productArea":"playbooks","score":8288.442181591929,"reasons":["search_match","title_match","display_title_match","term_match"],"markdown":"# Turning Expert Judgment into Policies and Rubrics\n\nUse this playbook when Expert Contribution notes are rich enough to explain correctness, but not yet structured enough to run repeatable benchmark checks."},{"blockId":"playbooks.expert-judgment-to-standards#related-docs","pageId":"playbooks.expert-judgment-to-standards","title":"Related docs","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#related-docs","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"related-docs","kind":"recipe","productArea":"playbooks","score":2703.779241512579,"reasons":["search_match","page_title_match","term_match"],"markdown":"## Related docs\n\n{% related-card-grid title=\"Related docs\" %}\n- [Writing Binary Rubrics](/docs/correctness-governance/binary-rubrics)\n- [Inspect Contributed Artifacts](/docs/expert-contributions/contributed-artifacts)\n- [Work with Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)\n- [Compare Harness Versions](/docs/benchmark-evaluations/compare)\n- [Read run results](/docs/benchmark-evaluations/inspect-results)\n- [Run a benchmark](/docs/benchmark-evaluations/run-evaluation)\n- [Importing cases](/docs/operating-manual/import-and-prepare-cases)\n{% /related-card-grid %}"},{"blockId":"playbooks.expert-judgment-to-standards#evidence-to-collect","pageId":"playbooks.expert-judgment-to-standards","title":"Evidence to collect","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#evidence-to-collect","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"evidence-to-collect","kind":"recipe","productArea":"playbooks","score":2690.2559065517066,"reasons":["search_match","page_title_match","term_match"],"markdown":"## Evidence to collect\n\n- Reviewed Cases, responses, and attributable Contribution rationale used to derive standards.\n- Grouped obligations, prohibitions, exceptions, and source requirements.\n- Approved policies, applicability logic, and binary rubrics with version history.\n- Back-check evidence showing whether Rubrics reproduce the intended expert boundary.\n- Benchmark runs and comparison results that use the approved standards."},{"blockId":"playbooks.expert-judgment-to-standards#do-not-materialize","pageId":"playbooks.expert-judgment-to-standards","title":"Do not materialize","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#do-not-materialize","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"do-not-materialize","kind":"recipe","productArea":"playbooks","score":2654.266885831755,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Do not materialize\n\nKeep the input as Contribution evidence when experts remain divided, the source is missing, the rule applies only to one unusual Case, or the proposed Rubric adds assumptions the expert never made.\n\n{% example-demo title=\"Eligibility answer standards\" %}\nAcross several Contributions, benefits specialists repeatedly reject answers that infer account-specific eligibility without the required fields. The team materializes a Policy separating general rules from account determinations, defines its applicability, and creates a binary Rubric requiring either the necessary evidence or an explicit uncertainty statement. Back-checking succeeds on the reviewed Cases, the owner approves the pair, and later evaluation exposes confident unsupported claims as a precise Rubric failure.\n{% /example-demo %}"},{"blockId":"playbooks.expert-judgment-to-standards#source-confidence","pageId":"playbooks.expert-judgment-to-standards","title":"Source confidence","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#source-confidence","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"source-confidence","kind":"recipe","productArea":"playbooks","score":2636.777650493179,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Source confidence\n\nDoctrine-backed: the approved product model keeps expert provenance, governed approval, and evaluator evidence distinct. Linked code-backed pages define contributed artifacts, Policy and Rubric controls, versioning, and evaluation."},{"blockId":"playbooks.expert-judgment-to-standards#from-contribution-to-governed-standard","pageId":"playbooks.expert-judgment-to-standards","title":"From Contribution to governed standard","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#from-contribution-to-governed-standard","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"from-contribution-to-governed-standard","kind":"recipe","productArea":"playbooks","score":2629.681408371211,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## From Contribution to governed standard\n\n1. In **Contributed Artifacts**, select the attributable answers and Cases that express the same decision boundary.\n2. Separate the reusable behavioral rule from applicability, supporting rationale, Case-specific preference, and unresolved disagreement.\n3. Create or revise the Policy in **Correctness Governance** and link representative included and excluded Cases.\n4. Create one binary Rubric for each observable must-level requirement. Keep “insufficient evidence” distinguishable from behavioral failure where the evaluator supports it.\n5. Back-check the draft pair against the expert-reviewed Cases. A Rubric that cannot reproduce the intended boundary needs revision before approval.\n6. Have the accountable owner approve and version the governed objects.\n7. Create the Benchmark Version that uses the approved evaluator boundary, then evaluate saved Harness Versions."},{"blockId":"playbooks.expert-judgment-to-standards#materialization-trigger","pageId":"playbooks.expert-judgment-to-standards","title":"Materialization trigger","pageTitle":"Turning Expert Judgment into Policies and Rubrics","url":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics#materialization-trigger","markdownUrl":"https://teammately.ai/docs/playbooks/turning-expert-judgment-into-policies-and-rubrics.md","sectionId":"materialization-trigger","kind":"recipe","productArea":"playbooks","score":2621.968988022342,"reasons":["search_match","page_title_match","term_match"],"markdown":"## Materialization trigger\n\nUse this when several attributable expert answers reveal a recurring obligation, prohibition, exception, or source requirement. A single comment may justify investigation but should not silently become a reusable standard."},{"blockId":"concepts.policies-rubrics#policies-and-rubrics","pageId":"concepts.policies-rubrics","title":"Policies and rubrics","pageTitle":"Policies and rubrics","url":"https://teammately.ai/docs/concepts/policies-and-rubrics.md","humanUrl":"https://teammately.ai/docs/concepts/policies-and-rubrics#policies-and-rubrics","markdownUrl":"https://teammately.ai/docs/concepts/policies-and-rubrics.md","sectionId":"policies-and-rubrics","kind":"concept","productArea":"object_model","score":2226.0901992175245,"reasons":["search_match","term_match"],"markdown":"# Policies and rubrics"}]}