Resolve conflicting correctness evidence
When a conflict needs resolution
Resolve a conflict when experts reach different conclusions from the same Case, when governed Policies contradict one another, when a Rubric tests a broader or narrower rule than its Policy, or when new source evidence changes the standard that earlier Benchmark Versions used.
Disagreement is not automatically a reviewer-quality problem. It often exposes missing context, mixed applicability, an unresolved source hierarchy, or two legitimate product boundaries that should be modeled separately.
Prerequisites
- Name the exact Case versions, outputs, Policies, Rubrics, expert responses, and sources in conflict.
- Preserve attribution and timestamps. Do not collapse opposing judgments into an unattributed summary.
- Separate factual disagreement from scope disagreement and from differences in desired product behavior.
- Identify the accountable owner for any governed object that may change.
Resolve a correctness conflict
- Open the affected governed object or Contribution and collect the linked Cases, expert rationale, source material, and activity history.
- Reconstruct each position in its strongest form: what evidence it uses, which situations it covers, and which outcome it recommends.
- Test whether the conflict disappears when applicability, target-system context, user segment, source authority, or time boundary is made explicit.
- If one position lacks required evidence, record that finding without erasing the original contribution.
- If both positions are valid in different contexts, split or refine the Policy, applicability, Rubric, Case, or coverage facet that conflated them.
- Have the accountable owner approve the resulting governed change. Expert participation alone does not approve it.
- Mark affected current evidence for follow-up, create new versions or a Snapshot where required, and preserve older Runs under their original boundary.
Object and state changes
- Correct the Case when required context or the judged output is wrong.
- Correct the Policy when the behavioral rule or its scope is wrong.
- Correct the Rubric when the test does not faithfully check the Policy.
- Correct coverage facets or membership when the benchmark over- or under-represents a boundary.
- Create a new Benchmark Version when the governed evaluation boundary changes.
- Keep an unresolved observation explicit when the source evidence cannot yet support a decision.
Success criteria
- A reviewer can see the original positions and the evidence behind each.
- The resolution names the artifact and version that changed.
- Approval authority is explicit.
- Downstream Case selection, standards, Snapshots, Runs, or customer-owned human review context are either still valid under a named boundary or routed for refresh.
- The team did not manufacture agreement by deleting dissenting evidence.
Common failure modes
- Voting before reconstructing the evidence and applicability behind each position.
- Editing a downstream Rubric when the conflict belongs to a Case or Policy boundary.
- Treating expert participation as approval of a governed object.
- Erasing dissent or historical versions after a resolution is approved.
Worked example
Two valid refund rules
One specialist rejects every refund exception; another approves exceptions for enterprise accounts. Their Cases reveal that both followed different authoritative programs. The team adds an account-program applicability boundary, revises the Policy and linked Rubrics, records the approval in activity history, and creates a new Benchmark Version. The earlier expert responses remain attributable evidence for why the split was needed.
Source confidence
Code-backed: Policy and Rubric detail routes expose governed objects, linked Cases, approval, and activity history, while Contribution review results preserve attributable expert learning. The evidence-reconciliation method is doctrine-backed; no single product screen automatically adjudicates every cross-object conflict.