Teammately Docs
Docs menu

recipe

Turning Expert Judgment into Policies and Rubrics

Convert specialist judgment into explicit, versioned, and testable correctness standards.

Turning Expert Judgment into Policies and Rubrics

Use this playbook when Expert Contribution notes are rich enough to explain correctness, but not yet structured enough to run repeatable benchmark checks.

Materialization trigger

Use this when several attributable expert answers reveal a recurring obligation, prohibition, exception, or source requirement. A single comment may justify investigation but should not silently become a reusable standard.

From Contribution to governed standard

  1. In Contributed Artifacts, select the attributable answers and Cases that express the same decision boundary.
  2. Separate the reusable behavioral rule from applicability, supporting rationale, Case-specific preference, and unresolved disagreement.
  3. Create or revise the Policy in Correctness Governance and link representative included and excluded Cases.
  4. Create one binary Rubric for each observable must-level requirement. Keep “insufficient evidence” distinguishable from behavioral failure where the evaluator supports it.
  5. Back-check the draft pair against the expert-reviewed Cases. A Rubric that cannot reproduce the intended boundary needs revision before approval.
  6. Have the accountable owner approve and version the governed objects.
  7. Create the Benchmark Version that uses the approved evaluator boundary, then evaluate saved Harness Versions.

Do not materialize

Keep the input as Contribution evidence when experts remain divided, the source is missing, the rule applies only to one unusual Case, or the proposed Rubric adds assumptions the expert never made.

Worked example

Eligibility answer standards

Across several Contributions, benefits specialists repeatedly reject answers that infer account-specific eligibility without the required fields. The team materializes a Policy separating general rules from account determinations, defines its applicability, and creates a binary Rubric requiring either the necessary evidence or an explicit uncertainty statement. Back-checking succeeds on the reviewed Cases, the owner approves the pair, and later evaluation exposes confident unsupported claims as a precise Rubric failure.

Evidence to collect

  • Reviewed Cases, responses, and attributable Contribution rationale used to derive standards.
  • Grouped obligations, prohibitions, exceptions, and source requirements.
  • Approved policies, applicability logic, and binary rubrics with version history.
  • Back-check evidence showing whether Rubrics reproduce the intended expert boundary.
  • Benchmark runs and comparison results that use the approved standards.

Related docs

Task / Correctness GovernanceWrite Binary RubricsWrite atomic pass-or-fail criteria grounded in governed policies, applicable cases, and observable candidate behavior.Reference / Expert ContributionsContributed ArtifactsInspect policies, rubrics, cases, and coverage observations produced through attributable expert contribution work.Reference / Correctness GovernancePolicies and RubricsUnderstand the governed relationship between behavior policies, applicability, binary rubrics, linked cases, and expert provenance.Task / Benchmark EvaluationsCompare Harness VersionsCompare two or more saved Harness Versions in a symmetric evidence matrix across Cases, evaluators, and Coverage Facets.Task / Benchmark EvaluationsInspect Evaluation ResultsTrace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.Task / Operating ManualImport and prepare casesBring cases into the project, conform them to Project Input Schema, inspect materials, and prepare benchmark selection.

Source confidence

Doctrine-backed: the approved product model keeps expert provenance, governed approval, and evaluator evidence distinct. Linked code-backed pages define contributed artifacts, Policy and Rubric controls, versioning, and evaluation.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Correctness Elicitation Correctness Elicitation Turn tacit specialist judgment into attributable contributions, governed policies, applicability conditions, and binary rubrics. Expert Contributions Expert Contributions Coordinate benchmark-scoped expert work, attributable judgment, governed artifacts, and the decisions that move correctness forward. Correctness Governance Write Binary Rubrics Write atomic pass-or-fail criteria grounded in governed policies, applicable cases, and observable candidate behavior.
AI context