# Policies, rubric overlap, and applicability
Generated: 2026-11-13T08:01:26.483Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: workflow.troubleshooting.standards
title: Policies, rubric overlap, and applicability
summary: A requirement is too broad, duplicates another check, or applies to the wrong situations.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-11
canonical: /docs/troubleshooting/standards
---
# Policies, rubric overlap, and applicability
## Symptom
A requirement is too broad, duplicates another check, or applies to the wrong situations.
## Likely causes
| What you observe | Focused guide |
| --- | --- |
| A policy generalizes beyond the expert requirement | [Overly Broad Policies](/docs/troubleshooting/overly-broad-policies) |
| Several rubrics score the same requirement | [Overlapping Rubrics](/docs/troubleshooting/overlapping-rubrics) |
| A criterion is selected or skipped incorrectly | [Weak Applicability Logic](/docs/troubleshooting/weak-applicability-logic) |
| Proposed standards do not preserve the intended distinction | [Noisy AI Suggestions](/docs/troubleshooting/noisy-ai-suggestions) |
## Fix
Compare the criterion with the expert’s reasoning and the Cases that define its boundary. Clarify the requirement and applicability separately. Use the owning review controls to decide the revised standard.
Follow the focused guide for the actual state you observe; keep accepted records and explicit human decisions intact.
## Prevention
Preserve exact object and version identities, review the relevant source context, and inspect the accepted state before comparing outcomes.
## Related task pages
{% related-card-grid title="Return to the workflow" %}
- [Complete your first alignment loop](/docs/quickstart)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Diagnose the observed state" %}
- [Overly Broad Policies](/docs/troubleshooting/overly-broad-policies)
- [Overlapping Rubrics](/docs/troubleshooting/overlapping-rubrics)
- [Weak Applicability Logic](/docs/troubleshooting/weak-applicability-logic)
- [Noisy AI Suggestions](/docs/troubleshooting/noisy-ai-suggestions)
{% /related-card-grid %}
---
id: troubleshooting.overly-broad-policies
title: Overly Broad Policies
summary: Narrow policies that are too vague to guide review, rubrics, or human reviews.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/overly-broad-policies
---
# Overly Broad Policies
Use this when a policy is too general to guide review, rubric writing, or human reviews.
## Symptom
Reviewers agree with the policy in principle but cannot decide whether a specific output satisfies it. Rubrics derived from the policy become vague, noisy, or impossible to apply consistently.
## Likely causes
- The policy describes a value such as “be helpful” without a behavioral boundary.
- Several domain rules were combined into one policy.
- The policy lacks linked cases that show pass and fail examples.
- The policy was approved from one expert comment without broader review.
## Diagnostic checks
- Ask whether the policy can produce a binary rubric without adding new assumptions.
- Find linked cases that should pass and fail the policy.
- Compare the policy to existing standards for duplicate or conflicting scope.
- Review benchmark failures to see whether the policy fits the observed behavior.
## Fix
- Narrow the policy to one behavior boundary.
- Split unrelated requirements into separate policies.
- Add linked cases and applicability before approval.
- Use the Policy's own governed approval and version controls after revision. A Contribution Checkpoint confirms Contribution progress; it is not automatically Policy approval.
## Prevention
- Draft policies from repeated judgments, not one-off comments.
- Require applicability and examples before approval.
- Review policy overlap during standards maintenance.
- Version revisions so old benchmark results remain interpretable.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)
- [Lemon Elicit](/docs/concepts/correctness-elicitation)
- [Resolve conflicting correctness results](/docs/governance/conflict-resolution)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Policies](/docs/object-model/policies)
- [Policy Versioning](/docs/governance/policy-versioning)
- [Lemon Elicit](/docs/concepts/correctness-elicitation)
{% /related-card-grid %}
---
id: troubleshooting.overlapping-rubrics
title: Overlapping Rubrics
summary: Identify duplicated criteria and resolve contradictory judgments.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/overlapping-rubrics
---
# Overlapping Rubrics
Use this when multiple rubrics appear to fail the same behavior or produce contradictory judgments.
## Symptom
A single output triggers several rubric failures that all describe the same defect, or two rubrics disagree about whether the same behavior should pass. Review context becomes noisy because failures are counted more than once.
## Likely causes
- A new rubric was created without checking existing approved rubrics.
- One rubric tests several behaviors and overlaps narrower checks.
- Policy revisions changed the behavior boundary but old rubrics remained active.
- Applicability rules bind multiple rubrics to the same irrelevant cases.
## Diagnostic checks
- Group failures by output and identify rubrics that cite the same response passage or recorded action.
- Compare rubric pass/fail wording against linked policies.
- Check rubric versions and whether old versions are still bound to benchmark cases.
- Inspect applicability rules for rubrics that should be mutually exclusive.
## Fix
- Choose the Rubric definition that should govern future evaluations and revise or version the affected governed objects through their available controls. Do not claim a merge or retire action unless it is visible in the current surface.
- Split multi-behavior rubrics into one binary check each.
- Revise applicability so rubrics apply only to the intended cases.
- After approving changes to criteria or applicability, create a new Dataset Snapshot and Benchmark Version that capture the updated standards.
- Evaluate the intended saved Harness Versions against the new Benchmark Version's full planned Case population. Inspect the affected Cases and Rubrics in Results.
Existing Runs retain their original criteria. Repeating a Run against the old Benchmark Version continues to assess the earlier standard. See [Benchmark Versioning](/docs/governance/benchmark-versioning) for the boundary to establish before evaluation.
## Prevention
- Search existing rubrics before approving a new one.
- Require a passing and failing example for every rubric boundary.
- Review overlap during policy/rubric approval.
- Treat rubric edits as versioned changes that can affect benchmark comparisons.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)
- [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Rubrics](/docs/object-model/rubrics)
- [Rubric Versioning](/docs/governance/rubric-versioning)
- [Applicability logic](/docs/object-model/applicability-logic)
{% /related-card-grid %}
---
id: troubleshooting.weak-applicability-logic
title: Weak Applicability Logic
summary: Fix standards that are applied to the wrong cases or skipped where they matter.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/weak-applicability-logic
---
# Weak Applicability Logic
Use this when policies or rubrics fire on irrelevant cases or miss cases where they should apply.
## Symptom
Benchmark results show failures that reviewers consider irrelevant, or important cases skip the standards that should govern them. The issue appears as false positives, false negatives, or confusing policy-level result counts.
## Likely causes
- The applicability condition uses a broad keyword or metadata field that does not prove the behavior is in scope.
- Cases lack the metadata or source context the applicability rule depends on.
- A policy boundary changed but applicability was not revised.
- Conversation context or retrieved-source state is not represented in the case.
## Diagnostic checks
- Inspect included and excluded cases side by side.
- Identify the exact source signal the applicability rule depends on.
- Check whether case metadata, dimensions, or context fields are missing or stale.
- Review recent policy or rubric revisions for boundary changes.
## Fix
- Rewrite applicability around source-backed signals, not broad topic labels.
- Add missing metadata or context before relying on the rule.
- Create examples that should be included and excluded, then test the boundary.
- Approve and version the corrected governed objects, then create the Benchmark boundary and Runs needed to evaluate the revised applicability.
## Prevention
- Define applicability before benchmark runs, not after reading failures.
- Keep case metadata and dimensions current.
- Review applicability whenever a policy or rubric version changes.
- Keep included and excluded example Cases linked to the governed standard where the current surface supports them.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics)
- [Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Applicability logic](/docs/object-model/applicability-logic)
- [Policies](/docs/object-model/policies)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
{% /related-card-grid %}
---
id: troubleshooting.noisy-ai-suggestions
title: Noisy AI Suggestions
summary: Refine proposed artifacts by checking their purpose, context, and the decisions needed to adopt them.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-11
canonical: /docs/troubleshooting/noisy-ai-suggestions
---
# Noisy AI Suggestions
## Symptom
AI-assisted suggestions look polished, but reviewers cannot safely use them for the artifact they affect. The suggestion may sound plausible while missing source material, applying too broadly, duplicating an existing standard, or skipping the relevant state boundary.
For proposed standards and accountable review decisions, the risk is usually premature approval. For Comparison Directions, the risk is different: AI proposals remain in their suggestion thread until accepted or rejected. Review the proposal and its scope before applying it; accepting a direction is separate from approving any generated response.
## Likely causes
- The suggestion was generated without enough case context, reviewer rationale, or source material.
- The suggestion collapses policy, applicability, and rubric into one fluent paragraph.
- Existing standards already cover the behavior, but the suggestion does not detect overlap.
- The artifact is still draft or suggested, but a downstream page treats it as approved.
- An AI-suggested Comparison Direction overlaps user-defined directions or no longer matches current project learning.
## Diagnostic checks
1. Open the source cases, outputs, reviewer notes, or interview answers that motivated the suggestion.
2. Check whether the suggestion names a policy rule, an applicability boundary, or a rubric check clearly.
3. Compare it with existing approved policies and rubrics for overlap or contradiction.
4. For Comparison Directions, compare it with active and archived directions before deciding whether to keep it.
5. Confirm whether the artifact is draft, suggested, approved, rejected, stale, superseded, active, archived, or dismissed.
## Fix
For policies, rubrics, classifications, and review context, keep the suggestion as draft context until a human owner edits, rejects, or approves the specific artifact. Split broad standards suggestions into Policy, applicability and Rubric material before approval. Diagnose an incorrect automatic classification through its owning classification workflow; do not invent a separate approval queue for every label. If a benchmark or review note already used the suggestion as governed standards, correct the interpretation note and rerun only after the approved artifact boundary is clear.
For Comparison Directions, review the proposed content in its thread. Accept a useful proposal or reject an unsuitable one. Save or discard unsaved manual edits before accepting a proposal for the same direction. Manage an existing direction through its normal edit, pin or delete controls. If it shows **Potentially stale**, review current relevance before dismissing that advisory; it is separate from accepting a proposed change.
## Prevention
Require source cases and reviewer rationale before promoting generated text. Keep suggested and approved states visually and procedurally separate for standards. Review overlap with existing standards before accepting a generated policy or rubric, and review overlap with active or archived directions before keeping an AI-suggested Comparison Direction.
{% example-demo title="Refining an unclear refund rule" %}
Visible problem: An AI-assisted draft says support answers should "be careful with refunds."
Check: The reviewer opens the cases and sees the real issue is unsupported refund exceptions.
Correction: The team rewrites the draft into a policy about approved refund support, applicability for refund requests, and a rubric that fails unsupported exception promises.
Outcome: The draft becomes useful preparation, but benchmark use waits for human approval.
{% /example-demo %}
{% example-demo title="Example: noisy Comparison Direction" %}
Visible problem: Teammately suggests a direction for "more cautious answers," but the project already has a user-created direction for unsupported-claim refusal.
Check: The owner compares the proposed direction with the current project library.
Correction: The owner rejects the overlapping proposal or requests a narrower direction before accepting it.
Outcome: No approval queue is needed. The direction list stays useful, and generated cases still need normal review before benchmark use.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Lemon and Human Decisions](/docs/governance/human-approval-boundaries)
- [Comparison Directions](/docs/assets/comparison-directions)
- [Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)
- [Complete your first alignment loop](/docs/quickstart)
- [Choose your starting point](/docs)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Policies](/docs/object-model/policies)
- [Rubrics](/docs/object-model/rubrics)
- [Applicability logic](/docs/object-model/applicability-logic)
- [Cases](/docs/object-model/cases)
{% /related-card-grid %}