---
id: playbooks.expert-judgment-to-standards
title: Turning Expert Judgment into Policies and Rubrics
summary: Convert specialist judgment into explicit, versioned, and testable correctness standards.
kind: recipe
product_area: playbooks
status: stable
updated: 2026-08-23
canonical: /docs/playbooks/turning-expert-judgment-into-policies-and-rubrics
---

# Turning Expert Judgment into Policies and Rubrics

Use this playbook when Expert Contribution notes are rich enough to explain correctness, but not yet structured enough to run repeatable benchmark checks.

## Materialization trigger

Use this when several attributable expert answers reveal a recurring obligation, prohibition, exception, or source requirement. A single comment may justify investigation but should not silently become a reusable standard.

## From Contribution to governed standard

1. In **Contributed Artifacts**, select the attributable answers and Cases that express the same decision boundary.
2. Separate the reusable behavioral rule from applicability, supporting rationale, Case-specific preference, and unresolved disagreement.
3. Create or revise the Policy in **Correctness Governance** and link representative included and excluded Cases.
4. Create one binary Rubric for each observable must-level requirement. Keep “insufficient evidence” distinguishable from behavioral failure where the evaluator supports it.
5. Back-check the draft pair against the expert-reviewed Cases. A Rubric that cannot reproduce the intended boundary needs revision before approval.
6. Have the accountable owner approve and version the governed objects.
7. Create the Benchmark Version that uses the approved evaluator boundary, then evaluate saved Harness Versions.

## Do not materialize

Keep the input as Contribution evidence when experts remain divided, the source is missing, the rule applies only to one unusual Case, or the proposed Rubric adds assumptions the expert never made.

{% example-demo title="Eligibility answer standards" %}
Across several Contributions, benefits specialists repeatedly reject answers that infer account-specific eligibility without the required fields. The team materializes a Policy separating general rules from account determinations, defines its applicability, and creates a binary Rubric requiring either the necessary evidence or an explicit uncertainty statement. Back-checking succeeds on the reviewed Cases, the owner approves the pair, and later evaluation exposes confident unsupported claims as a precise Rubric failure.
{% /example-demo %}

## Evidence to collect

- Reviewed Cases, responses, and attributable Contribution rationale used to derive standards.
- Grouped obligations, prohibitions, exceptions, and source requirements.
- Approved policies, applicability logic, and binary rubrics with version history.
- Back-check evidence showing whether Rubrics reproduce the intended expert boundary.
- Benchmark runs and comparison results that use the approved standards.

## Related docs

{% related-card-grid title="Related docs" %}
- [Writing Binary Rubrics](/docs/correctness-governance/binary-rubrics)
- [Inspect Contributed Artifacts](/docs/expert-contributions/contributed-artifacts)
- [Work with Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)
- [Compare Harness Versions](/docs/benchmark-evaluations/compare)
- [Read run results](/docs/benchmark-evaluations/inspect-results)
- [Run a benchmark](/docs/benchmark-evaluations/run-evaluation)
- [Importing cases](/docs/operating-manual/import-and-prepare-cases)
{% /related-card-grid %}

## Source confidence

Doctrine-backed: the approved product model keeps expert provenance, governed approval, and evaluator evidence distinct. Linked code-backed pages define contributed artifacts, Policy and Rubric controls, versioning, and evaluation.