# Low expert agreement Generated: 2026-09-14T05:50:32.234Z Source build: local Canonical docs: https://teammately.ai/docs --- id: troubleshooting.low-expert-agreement title: Low expert agreement summary: Investigate disagreement as evidence about context, applicability, source authority, or unresolved product policy. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/low-expert-agreement --- # Low expert agreement ## Symptoms - Experts choose different answers for the same Case or Checkpoint. - Rationale shows different assumptions even when the selected answer matches. - Agreement is high on routine Cases and collapses on one boundary segment. - A Contribution completes, but its learning cannot be materialized into one defensible standard. ## Likely causes - Experts saw different Case versions, sources, or task context. - The Case or question leaves the required judgment ambiguous. - Reviewers use different applicability or source-authority assumptions. - The domain genuinely contains unresolved or context-dependent policy. ## Preserve disagreement before interpreting it Do not average away the responses or select the majority answer before reading rationale. Confirm that each response belongs to the intended expert, task, Case version, question, and source context. A routing mistake and a legitimate domain disagreement require different fixes. ## Diagnostic checks 1. Open the review or session detail and compare the exact prompt, Case context, available sources, response, rationale, and timestamps. 2. Check whether all experts saw the same version and whether any required context was missing. 3. Separate four causes: ambiguous Case, unclear question, different applicability assumptions, and conflicting domain judgment. 4. Segment the disagreement by source authority, user type, risk, product state, or other relevant coverage facet. 5. Ask whether the current Policy already resolves the question. If it does, determine whether the Policy or its applicability was invisible or unclear to reviewers. ## Fix - **Missing context:** correct or version the Case and repeat only the affected review. - **Unclear question:** rewrite the task so the requested judgment and evidence threshold are explicit. - **Different applicability:** refine the Policy boundary or split Cases into distinct contexts. - **Conflicting expert judgment:** route the evidence through conflict resolution and an accountable owner. - **Wrong expertise:** revise assignment criteria without deleting the attributable response. Do not treat consensus as approval. Even unanimous expert input becomes a governed Policy, Rubric, Case-scoped reference output, or Benchmark Dataset decision only through the owning workflow. ## Prevention Package the same visible evidence for every assigned expert, state the requested judgment precisely, and include applicability or source-authority context when it controls the answer. Preserve rationale so later disagreement can be diagnosed without repeating every Contribution. {% example-demo title="Example: agreement drops only for enterprise accounts" %} Three experts agree on standard refunds but split on enterprise exceptions. Session detail shows that two use a newer enterprise program while one follows the public policy. The operator records both sources, adds the account-program boundary, and routes a Policy revision for approval. The disagreement becomes useful evidence instead of a reviewer score. {% /example-demo %} ## Source confidence Code-backed: review logs, session detail, review-result normalization, and expert-review types preserve task, session, response, and rationale context. The causes of disagreement and the appropriate governed resolution require human interpretation. ## Related task pages {% related-card-grid title="Related workflows" %} - [Resolve conflicting correctness evidence](/docs/governance/conflict-resolution) - [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Approval History and Reviewer Activity](/docs/governance/approval-history-and-reviewer-activity) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Expert Contributions](/docs/expert-contributions) - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) - [Reviewer and Project Access](/docs/governance/reviewer-and-project-access) {% /related-card-grid %} --- id: governance.conflict-resolution title: Resolve conflicting correctness evidence summary: Reconcile disagreement without hiding the Cases, expert judgments, sources, or versions that produced it. kind: task product_area: governance status: stable updated: 2026-08-23 canonical: /docs/governance/conflict-resolution --- # Resolve conflicting correctness evidence ## When a conflict needs resolution Resolve a conflict when experts reach different conclusions from the same Case, when governed Policies contradict one another, when a Rubric tests a broader or narrower rule than its Policy, or when new source evidence changes the standard that earlier Benchmark Versions used. Disagreement is not automatically a reviewer-quality problem. It often exposes missing context, mixed applicability, an unresolved source hierarchy, or two legitimate product boundaries that should be modeled separately. ## Prerequisites - Name the exact Case versions, outputs, Policies, Rubrics, expert responses, and sources in conflict. - Preserve attribution and timestamps. Do not collapse opposing judgments into an unattributed summary. - Separate factual disagreement from scope disagreement and from differences in desired product behavior. - Identify the accountable owner for any governed object that may change. ### Task steps: Resolve a correctness conflict 1. Open the affected governed object or Contribution and collect the linked Cases, expert rationale, source material, and activity history. 2. Reconstruct each position in its strongest form: what evidence it uses, which situations it covers, and which outcome it recommends. 3. Test whether the conflict disappears when applicability, target-system context, user segment, source authority, or time boundary is made explicit. 4. If one position lacks required evidence, record that finding without erasing the original contribution. 5. If both positions are valid in different contexts, split or refine the Policy, applicability, Rubric, Case, or coverage facet that conflated them. 6. Have the accountable owner approve the resulting governed change. Expert participation alone does not approve it. 7. Mark affected current evidence for follow-up, create new versions or a Snapshot where required, and preserve older Runs under their original boundary. ## Object and state changes - Correct the **Case** when required context or the judged output is wrong. - Correct the **Policy** when the behavioral rule or its scope is wrong. - Correct the **Rubric** when the test does not faithfully check the Policy. - Correct **coverage facets or membership** when the benchmark over- or under-represents a boundary. - Create a new **Benchmark Version** when the governed evaluation boundary changes. - Keep an unresolved observation explicit when the source evidence cannot yet support a decision. ## Success criteria - A reviewer can see the original positions and the evidence behind each. - The resolution names the artifact and version that changed. - Approval authority is explicit. - Downstream Case selection, standards, Snapshots, Runs, or customer-owned human review context are either still valid under a named boundary or routed for refresh. - The team did not manufacture agreement by deleting dissenting evidence. ## Common failure modes - Voting before reconstructing the evidence and applicability behind each position. - Editing a downstream Rubric when the conflict belongs to a Case or Policy boundary. - Treating expert participation as approval of a governed object. - Erasing dissent or historical versions after a resolution is approved. {% example-demo title="Two valid refund rules" %} One specialist rejects every refund exception; another approves exceptions for enterprise accounts. Their Cases reveal that both followed different authoritative programs. The team adds an account-program applicability boundary, revises the Policy and linked Rubrics, records the approval in activity history, and creates a new Benchmark Version. The earlier expert responses remain attributable evidence for why the split was needed. {% /example-demo %} ## Source confidence Code-backed: Policy and Rubric detail routes expose governed objects, linked Cases, approval, and activity history, while Contribution review results preserve attributable expert learning. The evidence-reconciliation method is doctrine-backed; no single product screen automatically adjudicates every cross-object conflict. ## Related reference pages {% related-card-grid title="Related workflows" %} - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) - [Versions, staleness, and resolution](/docs/object-model/versions-staleness-and-resolution) - [Approval History and Reviewer Activity](/docs/governance/approval-history-and-reviewer-activity) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Diagnose disagreement" %} - [Low expert agreement](/docs/troubleshooting/low-expert-agreement) - [Unclear Cases](/docs/troubleshooting/unclear-cases) - [Overlapping Rubrics](/docs/troubleshooting/overlapping-rubrics) {% /related-card-grid %} --- id: expert-contributions.complete title: Complete an Expert Contribution summary: Work through form, chat, interview, case-review, and checkpoint tasks while keeping specialist judgment attributable. kind: task product_area: expert_contributions status: stable updated: 2026-08-22 canonical: /docs/expert-contributions/complete-contribution --- # Complete an Expert Contribution Complete a Contribution by following its prepared task sequence and making the requested specialist judgments from the evidence shown. The expert experience can adapt between structured forms, agent chat, interviews, case review, and checkpoints. ## Prerequisites - A valid Contribution link or authenticated expert entry point. - Access to the Contribution and its assigned tasks. - Enough source and case context to explain each answer. - A stable connection when the task uses realtime agent interaction. ## Steps 1. Open the Contribution and read its objective, selected cases, and expected components before answering. 2. Complete each task according to its type. Planned activities can be Case Review, Form, Chat, or Interview; Curation, Comparative, and Trajectory components shape the prepared work those activities present. 3. Use attachments and visible case materials as the evidence boundary. State uncertainty when the supplied material does not resolve the question. 4. At a checkpoint, inspect the proposed summary or artifact meaning. Checkpoints prepare and reconcile requirements, consolidator or Policy statements, interview requests or records, and Rubrics. Confirm only what matches your judgment; retry, revise, or leave unresolved anything that does not. 5. Continue through the task handoff until the Contribution reaches its final step. 6. Review the completion state. If the experience shows a waiting, retry, or synchronization state, do not assume the administrator has received final evidence until the product confirms it. ## Object and state changes Answers create durable task responses and can advance task sessions, checkpoints, handoffs, and Contribution status. Chat or interview activity can produce transcripts and structured learning. Case review can attach judgment to selected cases. Completion makes the contribution available for reconciliation and materialization but does not itself make every proposed artifact governed. Review tasks can be `PREPARING`, `BLOCKED`, `READY`, `IN_PROGRESS`, `COMPLETED`, `SKIPPED`, or `SUPERSEDED`. The expert runtime can be `PREPARING`, `READY`, `ACTIVE`, `FINAL_CHECKPOINT`, `COMPLETED`, or `EXHAUSTED`. Checkpoints can be preparing, ready, or reconciled, with individual requirements pending, retryable, materialized, empty, or failed. These layered states explain why a Contribution can be active while one task is blocked or a final checkpoint is still pending. ## Success criteria - Every answer addresses the Contribution objective and cites the visible evidence where needed. - Case-level judgments remain connected to the relevant case. - Checkpoints distinguish accepted, revised, and unresolved meaning. - The final state is visibly complete rather than inferred from navigation. - Uncertainty or source conflict remains explicit for the administrator. ## Common failure modes - Answering from private background without identifying that the supplied evidence is incomplete. - Treating an agent summary as accurate without checking the checkpoint. - Leaving a form or chat task in a local unsynchronized state. - Continuing after a stale task handoff instead of following the current Contribution route. - Assuming that completion directly changes policies, rubrics, cases, or coverage. {% example-demo title="Example: checkpoint correction" %} An interview summary says that every expired agreement should be ignored. The expert corrects the checkpoint: expired agreements may still be relevant when the current agreement explicitly incorporates them. The corrected statement remains attributable and prevents an overbroad policy from being materialized. {% /example-demo %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Expert Contributions](/docs/expert-contributions) - [Contributed Artifacts](/docs/expert-contributions/contributed-artifacts) - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Expert Contribution problems](/docs/troubleshooting/expert-contributions) - [Permissions](/docs/troubleshooting/permissions) - [Authentication](/docs/troubleshooting/authentication) {% /related-card-grid %} ## Source confidence Code-backed: the current expert experience supports form, chat, interview, case-review, checkpoint, completion, waiting, and task-handoff routes with durable command and reconciliation behavior. --- id: coverage.boundary-cases title: Boundary Cases summary: Use edge and ambiguous cases to sharpen policies, applicability logic, and rubrics. kind: concept product_area: coverage_engineering status: stable updated: 2026-09-07 canonical: /docs/coverage-engineering/boundary-cases --- # Boundary Cases ## Definition A Boundary Case sits near a meaningful change in expected behavior: a Policy exception, an applicability edge, conflicting source authority, missing information, an escalation threshold, or another condition where a small contextual change should alter the correct output or action. Boundary Cases are not simply unusual examples. They are deliberate tests of a named distinction in the benchmark's coverage and correctness model. ## Why it matters Ordinary Cases can make a weak rule appear reliable because they never force the team to decide where it stops applying. Boundary Cases expose vague Policy scope, overlapping Rubrics, missing exceptions, and unsupported assumptions before those problems are hidden inside aggregate results. ## Where it appears in the product Coverage Stories can name a boundary as a missing or thin behavior slice. Case Foundry or another sourcing workflow can prepare candidate Cases against that need. Case Review checks whether the input, materials, and contrast are realistic. If the expected judgment is unclear, route the selected Cases into an Expert Contribution instead of inventing the answer during case construction. ## Artifacts it affects A useful Boundary Case records the exact contextual feature that changes the expectation, the controlling source material, the relevant Coverage Facets, and the Policy or Rubric question it exercises. Admission to the current Benchmark Dataset is a separate selection decision; admission to a Snapshot is a later immutable evidence decision. ## Operational check Compare the Boundary Case with a nearby ordinary Case. The pair should differ in the intended condition, not in several unrelated dimensions. Confirm that specialists can explain why the expected behavior changes, that the Rubric tests one observable criterion, and that the Case materials make the boundary visible to the evaluated system. {% example-demo title="Current versus superseded source" %} Two Cases ask the same eligibility question. One contains only the current policy document; the other also contains a plausible superseded document with conflicting terms. Specialists confirm that the response must follow the controlling current source or state uncertainty. The pair becomes a Boundary Case set for source authority, and its Rubric fails unsupported blending rather than generic factual error. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Manage Coverage Stories](/docs/coverage-management/coverage-stories) - [Review prepared Cases](/docs/coverage-management/case-review) - [Applicability Logic](/docs/object-model/applicability-logic) - [Comparison Directions](/docs/assets/comparison-directions) - [Product quickstart](/docs/quickstart) - [Task index](/docs/operating-manual/task-index) {% /related-card-grid %} ## Source confidence Code-backed: Coverage Stories can represent named coverage needs, Case Review exposes prepared Cases and materials, and Case sourcing types preserve coverage targets and source context. Specialist authority and evaluator approval remain separate workflows.