# Troubleshooting Generated: 2026-09-13T04:41:05.235Z Source build: local Canonical docs: https://teammately.ai/docs --- id: troubleshooting.overview title: Troubleshooting summary: Diagnose common Teammately setup, upload, review, classification, and benchmark issues. kind: concept product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting --- # Troubleshooting Use Troubleshooting when the correctness loop produces a confusing state: cases will not import, outputs are missing, reviewers cannot proceed, policies apply too broadly, coverage is unbalanced, or a benchmark result changes for reasons the team cannot yet explain. ## What this area is Start from the visible symptom and identify the owning object before changing anything. A missing response, an unmatched external output row, a stale Dimension, and an overbroad Policy can all distort evidence, but each requires a different correction. When the cause is uncertain, collect the exact Project, Benchmark Version, saved Harness Version or imported output identity, affected Case IDs, and visible error state before editing Cases, standards, coverage, or versions. ## Decision checkpoint | Symptom | Diagnose first | Likely owner artifact | | --- | --- | --- | | Cases will not import | Upload format, queue state, required columns | Case import or upload queue | | A managed Run has no response | Run state, Harness Version, and Case result | Evaluation Run | | Imported responses are missing | Case IDs and output-only Run mapping | External output mapping | | Experts cannot proceed | Contribution assignment, access, or source context | Expert Contributions or permissions | | Policies apply too broadly | Applicability logic and rubric wording | Correctness Governance | | Coverage looks unbalanced | Dimensions, ontology values, and benchmark membership | Coverage Engineering | | Score changed unexpectedly | Case, output, standard, benchmark, and run metadata versions | Benchmark Evaluations and Governance | ## Who uses it AI engineers usually diagnose import, mapping, benchmark, and output problems. Review owners diagnose reviewer access, unclear cases, and disagreement. Product leads and accountable owners use troubleshooting notes to understand whether a failure is a model behavior issue, an artifact issue, a coverage issue, or an access issue. ## Artifacts created or changed Troubleshooting can lead to changes in Cases, imports, output mappings, Policies, applicability, Rubrics, Dimensions, current Dataset membership, Benchmark Versions, Run configuration, or Project membership. Record which object changed and why; do not manufacture an unsupported generic recovery-note object. Preserve historical Snapshots, Runs, governed versions, Contribution attribution, and activity already recorded by the owning surfaces. Fix current state through the normal object workflow. ## How to use the recovery library Start with the observed symptom. Do not immediately change standards, rerun benchmarks, or edit cases until the failure source is clear. Missing outputs, weak applicability logic, stale dimensions, and unbalanced coverage can all make benchmark evidence look wrong, but they require different fixes. After the immediate fix, preserve the lesson. A recovery path should make the next correctness loop stronger: clearer case context, tighter applicability, better output mapping, better benchmark coverage, or more explicit run metadata. If the reader cannot name the symptom yet, use the task index only after identifying the blocked artifact. Planned operations belong in the operating manual; ambiguous or broken states belong here first. ## Recovery proof | Symptom class | The issue is actually fixed when... | Keep diagnosing if... | | --- | --- | --- | | Case import or upload | The affected queue or case record is complete enough for preparation work. | Rows moved forward but required context or mapping is still missing. | | Managed Run response | The Run reaches a terminal state and each evaluable Case has the generated response and Rubric outcomes expected for that Run. | The response is absent, the attempt failed, or the Case remains unevaluable. | | Imported output mapping | Every intended external row maps to the immutable Case ID in one output-only Run. | Rows are unmatched, duplicated, or joined by position. | | Review blockage | The reviewer has access, source context, assignment state, and questions needed to judge. | The reviewer can enter the screen but cannot make an accountable decision. | | Weak applicability | The standard now applies to a named case boundary instead of broad intent. | The same failure could pass or fail depending on reviewer interpretation. | | Unexpected score change | The team can name whether candidate behavior, case membership, standards, outputs, or metadata changed. | The cause is still described as "the benchmark changed." | ## Common starting tasks - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Read run results](/docs/benchmark-evaluations/inspect-results) - [Run a benchmark](/docs/benchmark-evaluations/run-evaluation) - [Importing cases](/docs/operating-manual/import-and-prepare-cases) - [Agent Setup](/docs/agent-setup) ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Agent instructions](/docs/agent-instructions) - [Case versions](/docs/governance/case-versioning) - [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases) - [Reference and golden outputs](/docs/object-model/reference-and-golden-outputs) - [Input-only and output-attached cases](/docs/assets/cases) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Access troubleshooting](/docs/troubleshooting/authentication) - [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs) - [Case upload troubleshooting](/docs/troubleshooting/dataset-upload) - [Permissions troubleshooting](/docs/troubleshooting/permissions) - [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} {% example-demo title="Unexpected benchmark change" %} A score changes after a Dataset refresh even though the saved Harness Version is unchanged. The team confirms that the second Run used a new Benchmark Version with additional boundary Cases. It treats the result as evidence under an expanded benchmark, not as a candidate regression, and compares Case-level Rubric outcomes within each named boundary. {% /example-demo %} ## Source confidence Code-backed: the cited routes cover the principal access, membership, Case upload, output mapping, coverage, and Policy surfaces routed from this index. Each linked troubleshooting page narrows its own claims to the current owning implementation. --- id: troubleshooting.authentication title: IP access restriction summary: Resolve the current Access Restricted page when the detected IP is not on the Workspace allowlist. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/authentication --- # IP access restriction Use this when Teammately displays **Access Restricted** and says the current IP is not on the Workspace allowlist. Sign-in, Project membership, and Contribution assignment are separate problems. ## Symptom The page reports **Access restricted by IP** and shows a detected client-header IP, backend-detected IP, or both. ## Likely causes - The current IPv4 or IPv6 address is not on the Workspace allowlist. - A VPN, proxy, gateway, or network change caused the detected address to differ from the expected address. - The client-header and backend-detected addresses differ, so the administrator needs the value actually shown by the product. ## Diagnostic checks - Read both detected IP values shown on the page. - Confirm whether the user recently changed networks or enabled a VPN. - Copy the displayed address instead of relying on an address from another device or external lookup. - If the page no longer appears but Project content is still unavailable, continue with permissions troubleshooting. ## Fix - Send the displayed IP to the Workspace administrator. - Have the administrator add the required single address or CIDR in the applicable security settings. - Retry after the allowlist change is active. If the detected address changed again, resolve the network or VPN source before adding more entries. ## Prevention - Confirm the expected access network before time-sensitive Contributions or evaluations. - Prefer a stable organizational network path when the Workspace uses IP restrictions. - Keep IP restriction, Project membership, and Contribution assignment diagnoses separate. ## Related task pages {% related-card-grid title="Related task pages" %} - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) - [Reviewing cases](/docs/expert-contributions/complete-contribution) - [Product quickstart](/docs/quickstart) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Permissions](/docs/reference/permissions) - [Reviewer and project access](/docs/governance/reviewer-and-project-access) - [Workspaces, projects, and target systems](/docs/object-model/workspaces-projects-and-target-systems) {% /related-card-grid %} ## Source confidence Code-backed: the Access Restricted route displays the client-header and backend-detected IP values, provides copy controls, and directs the user to an administrator for allowlisting. This page does not claim a broader sign-in or authentication contract. --- id: troubleshooting.benchmark-results-changed title: Benchmark Results Changed Unexpectedly summary: Diagnose result changes across target behavior, benchmark cases, standards, and versions. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/benchmark-results-changed-unexpectedly --- # Benchmark Results Changed Unexpectedly Use this when benchmark results change and the team is not sure whether the cause is model behavior or an artifact change. ## Symptom A pass rate, policy-level result, rubric result, or case-level outcome changes between runs even though the expected candidate behavior did not obviously change. ## Likely causes - Benchmark membership changed through case import, promotion, removal, or refresh. - Policy, rubric, or applicability versions changed between runs. - A different saved Harness Version, execution setting, or imported output-only Run was inspected. - Run metadata or benchmark version selection differs from the previous run. ## Diagnostic checks - Compare benchmark version, case count, policy versions, rubric versions, and applicability boundary. - Inspect changed Cases and confirm whether each result came from a managed Run or imported output-only Run. - Check run metadata for candidate, prompt, retrieval, or model differences. - Open policy/rubric result changes and trace them to exact cases. ## Fix - If artifacts changed, label the comparison as an artifact-boundary change rather than a pure behavior regression. - If the candidate identity or settings changed, run the intended saved Harness Version with the intended settings. - If the Benchmark boundary changed, create and name the appropriate Snapshot and Benchmark Version; do not rewrite the older Run. - If the cause remains unclear, hold downstream action until the changed evidence can be explained. ## Prevention - Record benchmark, case, policy, rubric, and candidate versions for every run. - Use comparison views before summarizing score movement. - Treat mapping, coverage, and standard changes as review-context boundaries. - Keep previous runs reproducible for audit. ## Related task pages {% related-card-grid title="Related task pages" %} - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Read run results](/docs/benchmark-evaluations/inspect-results) - [Inspect execution settings](/docs/benchmark-evaluations/execution-settings) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Versions, staleness, and resolution](/docs/object-model/versions-staleness-and-resolution) - [Benchmark Evaluations](/docs/benchmark-evaluations) - [Benchmark versioning](/docs/governance/benchmark-versioning) {% /related-card-grid %} ## Source confidence Code-backed: Dataset Snapshots, Compare, Run detail, and Run Metadata expose the identities and boundaries needed to separate Dataset, candidate, configuration, and imported-output changes. --- id: troubleshooting.benchmark-runs title: Benchmark run troubleshooting summary: Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/benchmark-runs --- # Benchmark run troubleshooting ## Symptoms - The evaluation surface has no Benchmark Version to run. - Output import or mapping cannot identify a Case or output column. - A Run is created but remains empty, incomplete, or failed. - Results appear under the wrong candidate label or Harness Version. - Two Runs show a score change but do not share a comparable evidence boundary. ## Likely causes - No immutable Snapshot exists for the intended evidence set. - Imported outputs are mapped to the wrong Case or column. - The saved Harness Version or candidate metadata does not match the evaluated system. - The Run is partial, failed, or being compared across different Benchmark Versions. ## Check the boundary before the failure 1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version. 2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run. 3. In Run setup, confirm the saved Harness Version and candidate metadata describe the system that produced the outputs. 4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior. 5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates. ## Fix - **No runnable version:** finish Case selection and create a Snapshot first. - **No mapped outputs:** correct Case identifiers or column mapping, then submit again under the intended Run. - **Wrong Harness or metadata:** create a correctly configured Run. Do not relabel completed evidence to represent a different system. - **Partial failure:** preserve successful per-Case evidence when the product does, correct the failed input or execution boundary, and rerun using a clearly named attempt. - **Confusing comparison:** compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed. ## Prevention Create a Snapshot before execution, save the exact Harness Version, preview output mapping on representative Cases, and name candidate metadata consistently. Before comparing, confirm that every difference between the two Runs is intentional and visible. ## Recovery check Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping. {% example-demo title="Example: scores drop after output import" %} A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version. {% /example-demo %} ## Source confidence Code-backed: Runs index, setup, output mapping, Run detail, and metadata-display implementations establish the identifiers, mapping choices, and evidence shown during diagnosis. Backend-provider errors and customer Harness behavior may require additional operational logs outside this page. ## Related task pages {% related-card-grid title="Related workflows" %} - [Run an evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect results](/docs/benchmark-evaluations/inspect-results) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Run Metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Benchmarks and versions](/docs/concepts/benchmarks-and-versions) - [Benchmark Snapshots](/docs/coverage-engineering/benchmark-snapshots) - [Outputs](/docs/object-model/outputs) {% /related-card-grid %} --- id: troubleshooting.dataset-upload title: Case upload troubleshooting summary: Repair uploads with rejected rows, missing inputs, incorrect column mapping, or Cases that arrive without usable context. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/dataset-upload --- # Case upload troubleshooting ## Symptoms - The file is rejected before column mapping. - Required input fields are absent or cannot be selected. - The preview puts outputs, metadata, or conversation context in the wrong fields. - The upload completes but expected Cases are not visible in the Case Pool. - Imported Cases are visible but cannot be reviewed or classified reliably. ## Likely causes - The source file has unusable headers, duplicate columns, or blank input rows. - Required Case input is unmapped or mapped from the wrong column. - Conversation context, output identity, or metadata was flattened into the wrong field. - The import completed, but the operator expected candidate Cases to become Benchmark members automatically. ## Diagnose the upload in order 1. Confirm the selected file is the intended source and uses a supported tabular shape for the upload surface. 2. Inspect headers and several real rows outside Teammately. Remove blank header rows, merged presentation cells, duplicate column names, and records that contain no usable input. 3. Return to **Map columns**. Map the Case input first, then candidate or reference output only when that identity is known. Map contextual columns as context or metadata rather than concatenating them into the input. 4. Review the preview for nulls, truncated objects, serialized JSON, and swapped user/assistant fields. 5. Submit a small representative file before importing the whole source. 6. After completion, open the Case Pool and verify row counts, a simple Case, a boundary Case, a conversation Case, and a row with missing optional data. ## Fix - **No required input:** choose the correct input column or add a normalized input field to the source file. - **Conversation collapsed into text:** preserve the structured turn context supported by the Project instead of guessing a public conversation schema. - **Output mapped as input:** correct the mapping and re-upload; do not silently edit already selected benchmark evidence. - **Missing context:** map supporting columns, then decide whether the existing imported Cases should be superseded or retained as an identifiable attempt. - **Duplicate Cases:** compare stable source identifiers and Case content before adding candidates to a Benchmark. ## Prevention Keep an untouched source file, a documented mapping decision, and a small QA sample. Treat database or internal representation details as implementation, not as the upload contract. The product preview is the last checkpoint before rows become Case Pool candidates. {% example-demo title="Example: transcripts import without account tier" %} The upload succeeds, but reviewers cannot explain why enterprise exceptions apply. The operator finds that `account_tier` was left unmapped. They add it as context, re-upload a representative sample, confirm the field in Case presentation, and only then import the full set. The incomplete candidates are not added to a Benchmark. {% /example-demo %} ## Source confidence Code-backed: the Case Pool upload, column-mapping, queue, and Case Pool surfaces establish file selection, mapping, submission state, and post-upload visibility. Supported customer file details remain whatever the current UI validates; this page does not create a public ingestion schema. ## Related task pages {% related-card-grid title="Related workflows" %} - [Import and prepare Cases](/docs/operating-manual/import-and-prepare-cases) - [Case Pool](/docs/coverage-engineering/case-pool) - [Metadata and context](/docs/reference/metadata-and-context) - [Unclear Cases](/docs/troubleshooting/unclear-cases) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Cases](/docs/object-model/cases) - [Case Pool](/docs/object-model/case-pool) - [Metadata and context](/docs/reference/metadata-and-context) {% /related-card-grid %} --- id: troubleshooting.dimension-classification title: Dimension classification troubleshooting summary: Resolve missing, ambiguous, or inconsistent Case classifications before they distort coverage analysis. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/dimension-classification --- # Dimension classification troubleshooting ## Symptoms - Many Cases show no value for an important Dimension. - Similar Cases receive different ontology values without a clear reason. - One Case appears to belong to several mutually exclusive values. - A coverage gap disappears or appears after labels change, although the Case set did not. - Reviewers cannot tell whether **unknown**, **not applicable**, and missing classification mean different things. ## Likely causes - The Dimension definition or ontology values overlap. - Required Case context is absent from the classification view. - The schema changed after existing Cases were classified. - The Dimension bundles independent behavior axes or leaves absence undefined. ## Diagnose the schema before the Cases 1. Open the Dimension and read its definition, ontology values, examples, and origin. 2. Decide whether the values are intended to be mutually exclusive, multi-label, ordered, or merely descriptive. Do not infer this from label names alone. 3. Sample Cases from each value plus unclassified Cases. Compare the full Case context, not only the short input shown in a table. 4. Look for overlapping definitions, missing fallback treatment, context fields unavailable to classification, or a Dimension that bundles more than one behavior axis. 5. Check whether the Dimension or ontology changed after the Cases were classified. ## Fix - Correct an individual Case classification when the schema is clear and the Case was mislabeled. - Improve the Dimension definition or ontology descriptions when reviewers interpret them differently. - Split a Dimension when one label depends on two independent behavior axes. - Add an explicit unknown or not-applicable treatment when absence carries meaning. - Reclassify affected Cases after a schema change, then review coverage plans and Snapshot boundaries before relying on segment results. Do not repair a misleading coverage chart by editing counts or selecting convenient Cases. Correct the Dimension or classification state that produced it. ## Prevention Define the classification rule and unknown treatment before broad use, attach representative examples to each value, and test boundary Cases with more than one informed reviewer. Review affected classifications whenever the Dimension schema changes. ## Verification Have two informed reviewers classify a small boundary sample using only documented context. Agreement is evidence that the schema is usable; disagreement should produce a clearer definition, better context, or an explicit unresolved boundary—not forced consensus. {% example-demo title="Example: overlapping source-support values" %} Cases alternate between **implied support** and **conflicting source** because one document implies compatibility while another denies it. The team clarifies that any authoritative contradiction uses **conflicting source**, adds examples, and reclassifies the affected Cases. They review the benchmark plan before creating a new Snapshot. {% /example-demo %} ## Source confidence Code-backed: Dimension list/detail surfaces, record-table classification utilities, and classification types establish how Dimension values appear on Cases. The product cannot determine whether a Project's vocabulary is conceptually sound without human review. ## Related task pages {% related-card-grid title="Related workflows" %} - [Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) - [Generate a dimension schema](/docs/coverage-engineering/generate-dimension-schema) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Stale Dimensions](/docs/troubleshooting/stale-dimensions) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Ontology](/docs/object-model/ontology) - [Metadata and context](/docs/reference/metadata-and-context) {% /related-card-grid %} --- id: troubleshooting.expert-contributions title: Expert Contribution problems summary: Diagnose contribution access, task routing, checkpoint, synchronization, completion, and artifact-reconciliation problems. kind: error product_area: troubleshooting status: stable updated: 2026-08-22 canonical: /docs/troubleshooting/expert-contributions --- # Expert Contribution problems ## Symptom An expert cannot open the Contribution, reaches the wrong task, sees stale work, cannot submit a response, remains at a checkpoint or waiting state, or completes the experience without the administrator seeing reconciled evidence. ## Likely causes - The expert lacks access or opened a link for a different Contribution. - The task handoff changed and the browser still points to an earlier route. - A form, chat, interview, or case-review command has not synchronized. - The task lease, realtime connection, or retry state needs recovery. - A checkpoint still requires confirmation or correction. - The expert experience is complete, but contributed artifacts have not yet been reconciled in the administrator workspace. - The Contribution was cancelled or changed while the expert was working. ## Fix 1. Confirm the Contribution identity and expert assignment in **Expert Contributions → Contributions**. 2. Inspect its current status and task plan rather than relying on the expert's previous URL. 3. Ask the expert to reopen the Contribution entry route so the current task handoff can resolve. 4. If the UI shows a waiting, retry, or synchronization state, follow that state and avoid duplicate submissions. 5. Inspect checkpoints for an unconfirmed or corrected statement. 6. Use **Logs & Status** to distinguish expert completion from reconciliation or artifact-materialization work. 7. Check **Contributed Artifacts** and the owning Correctness Governance, Assets, or Coverage Management surface before concluding that accepted material is missing. ## Prevention - Keep Contribution objectives and task sequences narrow. - Verify expert access before sending the request. - Preserve checkpoints for consequential meaning. - Use the current Contribution entry route instead of bookmarking an internal task route. - Teach administrators that task completion, evidence reconciliation, and artifact governance are separate transitions. - Avoid submitting the same response repeatedly when the product shows a pending or retryable synchronization state. {% example-demo title="Example: completed task but missing policy" %} An expert completes the final checkpoint, but the policy does not yet appear in Correctness Governance. Logs & Status shows that the Contribution is complete while evidence reconciliation remains active. The administrator waits for reconciliation, then inspects Contributed Artifacts and materializes the policy through its owning workflow instead of asking the expert to repeat the task. {% /example-demo %} ## Related task pages {% related-card-grid title="Related task pages" %} - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) - [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution) - [Product quickstart](/docs/quickstart) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Expert Contributions](/docs/expert-contributions) - [Contributed Artifacts](/docs/expert-contributions/contributed-artifacts) - [Roles and permissions](/docs/reference/permissions) {% /related-card-grid %} ## Source confidence Code-backed: the expert routes, contribution runtime, durable command synchronization, and administrator status surfaces support these diagnostic boundaries. --- id: troubleshooting.low-expert-agreement title: Low expert agreement summary: Investigate disagreement as evidence about context, applicability, source authority, or unresolved product policy. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/low-expert-agreement --- # Low expert agreement ## Symptoms - Experts choose different answers for the same Case or Checkpoint. - Rationale shows different assumptions even when the selected answer matches. - Agreement is high on routine Cases and collapses on one boundary segment. - A Contribution completes, but its learning cannot be materialized into one defensible standard. ## Likely causes - Experts saw different Case versions, sources, or task context. - The Case or question leaves the required judgment ambiguous. - Reviewers use different applicability or source-authority assumptions. - The domain genuinely contains unresolved or context-dependent policy. ## Preserve disagreement before interpreting it Do not average away the responses or select the majority answer before reading rationale. Confirm that each response belongs to the intended expert, task, Case version, question, and source context. A routing mistake and a legitimate domain disagreement require different fixes. ## Diagnostic checks 1. Open the review or session detail and compare the exact prompt, Case context, available sources, response, rationale, and timestamps. 2. Check whether all experts saw the same version and whether any required context was missing. 3. Separate four causes: ambiguous Case, unclear question, different applicability assumptions, and conflicting domain judgment. 4. Segment the disagreement by source authority, user type, risk, product state, or other relevant coverage facet. 5. Ask whether the current Policy already resolves the question. If it does, determine whether the Policy or its applicability was invisible or unclear to reviewers. ## Fix - **Missing context:** correct or version the Case and repeat only the affected review. - **Unclear question:** rewrite the task so the requested judgment and evidence threshold are explicit. - **Different applicability:** refine the Policy boundary or split Cases into distinct contexts. - **Conflicting expert judgment:** route the evidence through conflict resolution and an accountable owner. - **Wrong expertise:** revise assignment criteria without deleting the attributable response. Do not treat consensus as approval. Even unanimous expert input becomes a governed Policy, Rubric, Case-scoped reference output, or Benchmark Dataset decision only through the owning workflow. ## Prevention Package the same visible evidence for every assigned expert, state the requested judgment precisely, and include applicability or source-authority context when it controls the answer. Preserve rationale so later disagreement can be diagnosed without repeating every Contribution. {% example-demo title="Example: agreement drops only for enterprise accounts" %} Three experts agree on standard refunds but split on enterprise exceptions. Session detail shows that two use a newer enterprise program while one follows the public policy. The operator records both sources, adds the account-program boundary, and routes a Policy revision for approval. The disagreement becomes useful evidence instead of a reviewer score. {% /example-demo %} ## Source confidence Code-backed: review logs, session detail, review-result normalization, and expert-review types preserve task, session, response, and rationale context. The causes of disagreement and the appropriate governed resolution require human interpretation. ## Related task pages {% related-card-grid title="Related workflows" %} - [Resolve conflicting correctness evidence](/docs/governance/conflict-resolution) - [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Approval History and Reviewer Activity](/docs/governance/approval-history-and-reviewer-activity) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Expert Contributions](/docs/expert-contributions) - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) - [Reviewer and Project Access](/docs/governance/reviewer-and-project-access) {% /related-card-grid %} --- id: troubleshooting.missing-outputs title: Missing Outputs summary: Separate a missing managed Run response from an unmatched row in an imported output-only Run. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/missing-outputs --- # Missing Outputs Use this when an Evaluation Run has no response for an intended Case. First identify whether the evidence came from managed Harness execution or an imported output-only Run. ## Symptom A managed Run is incomplete or has a failed or unevaluable Case attempt, or an imported output-only Run shows an unmatched Case row. An input-only Case by itself is not an error: managed execution is expected to generate its response. ## Likely causes - Managed execution failed, stopped, or could not evaluate the Case. - The wrong saved Harness Version or execution settings were selected. - An external output row has a missing, unknown, or duplicate immutable Case ID. - The imported file omitted Cases that the output-only Run was intended to cover. ## Diagnostic checks - Open the Run and check its lifecycle, attempts, failed Cases, and selected Harness Version. - Confirm that the Case belongs to the immutable Benchmark Version used by the Run. - For imported outputs, compare external identifiers with the immutable Case IDs shown by the mapping surface. - Confirm that the intended output column and candidate metadata were selected before import. ## Fix - For managed execution, correct the Harness or execution problem and start a new Run against the same Benchmark Version when that boundary is still intended. - For imported outputs, correct the external rows and repeat mapping with immutable Case IDs. - Keep managed and imported-output diagnoses separate; an imported response does not become a saved Harness Version. - Change the current Dataset and create a new Snapshot only when Case membership itself is wrong—not merely because a Run response is missing. ## Prevention - Save the exact Harness Version before managed execution and review execution settings. - Export or preserve immutable Case IDs through external generation. - Validate several mapped rows before creating an output-only Run. - Use Run Metadata to identify the candidate or external output set without pretending it is a Harness Version. ## Related task pages {% related-card-grid title="Related task pages" %} - [Import and prepare cases](/docs/operating-manual/import-and-prepare-cases) - [Run a benchmark](/docs/benchmark-evaluations/run-evaluation) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Target model outputs](/docs/object-model/outputs) - [Input-only and output-attached cases](/docs/assets/cases) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %} ## Source confidence Code-backed: the Run modal and Run detail establish managed execution identity and status; the output-mapping modal and List surface establish imported output-only Runs and unmatched Case handling. Input-only Cases are valid inputs to managed execution. --- id: troubleshooting.noisy-ai-suggestions title: Noisy AI Suggestions summary: Triage generated suggestions that are fluent but not ready for the artifact they affect. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/noisy-ai-suggestions --- # Noisy AI Suggestions ## Symptom AI-assisted suggestions look polished, but reviewers cannot safely use them for the artifact they affect. The suggestion may sound plausible while missing source evidence, applying too broadly, duplicating an existing standard, or skipping the relevant state boundary. For policies, rubrics, classifications, and review context, the risk is usually premature approval. For Comparison Directions, the risk is different: AI-suggested directions are active rows, so a noisy suggestion should be edited, archived, deleted, or reviewed through the normal direction controls rather than routed through a separate approval queue. ## Likely causes - The suggestion was generated without enough case context, reviewer rationale, or source material. - The suggestion collapses policy, applicability, and rubric into one fluent paragraph. - Existing standards already cover the behavior, but the suggestion does not detect overlap. - The artifact is still draft or suggested, but a downstream page treats it as approved. - An AI-suggested Comparison Direction overlaps user-defined directions or no longer matches current project learning. ## Diagnostic checks 1. Open the source cases, outputs, reviewer notes, or interview answers that motivated the suggestion. 2. Check whether the suggestion names a policy rule, an applicability boundary, or a rubric check clearly. 3. Compare it with existing approved policies and rubrics for overlap or contradiction. 4. For Comparison Directions, compare it with active and archived directions before deciding whether to keep it. 5. Confirm whether the artifact is draft, suggested, approved, rejected, stale, superseded, active, archived, or dismissed. ## Fix For policies, rubrics, classifications, and review context, keep the suggestion as draft context until a human owner edits, rejects, or approves the specific artifact. Split broad suggestions into policy, applicability, and rubric material before approval. If a benchmark or review note already used the suggestion as governed evidence, correct the evidence note and rerun only after the approved artifact boundary is clear. For Comparison Directions, handle the suggestion as a normal active direction. Edit it if the boundary is useful but phrased poorly, archive or delete it if it is not useful, and dismiss **Potentially stale** only when the team decides the direction still applies. ## Prevention Require source cases and reviewer rationale before promoting generated text. Keep suggested and approved states visually and procedurally separate for standards. Review overlap with existing standards before accepting a generated policy or rubric, and review overlap with active or archived directions before keeping an AI-suggested Comparison Direction. {% example-demo title="Noisy AI Suggestions boundary" %} Visible problem: An AI-assisted draft says support answers should "be careful with refunds." Check: The reviewer opens the cases and sees the real issue is unsupported refund exceptions. Correction: The team rewrites the draft into a policy about approved refund support, applicability for refund requests, and a rubric that fails unsupported exception promises. Outcome: The draft becomes useful preparation, but benchmark evidence waits for human approval. {% /example-demo %} {% example-demo title="Example: noisy Comparison Direction" %} Visible problem: Teammately suggests a direction for "more cautious answers," but the project already has a user-created direction for unsupported-claim refusal. Check: The owner compares the AI-suggested row with current and archived Comparison Directions. Correction: The owner archives the overlapping AI direction or edits it into a narrower boundary. Outcome: No approval queue is needed. The direction list stays useful, and generated cases still need normal review before benchmark use. {% /example-demo %} ## Related task pages {% related-card-grid title="Related task pages" %} - [What AI Features Can and Cannot Do](/docs/governance/what-ai-features-can-and-cannot-do) - [Human Approval Boundaries](/docs/governance/human-approval-boundaries) - [Comparison Directions](/docs/assets/comparison-directions) - [Editing Suggested Rubrics](/docs/correctness-governance/policies-and-rubrics) - [Product quickstart](/docs/quickstart) - [Task index](/docs/operating-manual/task-index) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Policies](/docs/object-model/policies) - [Rubrics](/docs/object-model/rubrics) - [Applicability logic](/docs/object-model/applicability-logic) - [Cases](/docs/object-model/cases) {% /related-card-grid %} ## Source confidence Code-backed: Dimension proposals, synthesized Case cards, Comparison Direction controls, and Policy approval expose distinct generated-artifact lifecycles. The diagnostic advice follows those visible boundaries; it does not claim that every AI-assisted surface uses the same approval state. --- id: troubleshooting.output-mapping title: Output mapping troubleshooting summary: Fix external output rows that do not join cleanly to immutable Case IDs in an output-only Run. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/output-mapping --- # Output mapping troubleshooting Use this when externally produced responses exist but the mapping preview cannot join them cleanly to the Cases in the selected Benchmark Version. ## Symptom Rows are unmatched, duplicated, or mapped to the wrong Case ID, or the wrong column was selected as the response. No output-only Run should be trusted until the mapping preview is correct. ## Likely causes - External rows use row order or display text instead of immutable Case IDs. - The file contains Case IDs from another Benchmark Version or Dataset export. - The selected response column or identifier column is wrong. - Several candidate output sets were combined without distinct metadata. ## Diagnostic checks - Compare several affected case IDs with the output source IDs. - Check whether the output set has candidate metadata such as model, prompt, retrieval version, or run identity. - Confirm the source file was produced from the same immutable Case population. - Inspect duplicate, blank, and unknown identifier counts in the mapping preview. ## Fix - Correct the external file and map using immutable Case IDs rather than row position. - Select the intended response column and keep one candidate output set per output-only Run. - Create a new output-only Run from the corrected mapping. Existing Runs remain evidence under their original mapping. - Repair missing Case context through the Case workflow, not during output mapping; create a new version boundary when Case content changes. ## Prevention - Preserve source IDs through import and output generation. - Sample mapped outputs before expert contribution work and before Evaluation Runs. - Keep candidate identity in run metadata. - Treat each imported output-only Run as its own evidence record; it is not a saved Harness Version. ## Related task pages {% related-card-grid title="Related task pages" %} - [Importing cases](/docs/operating-manual/import-and-prepare-cases) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Target model outputs](/docs/object-model/outputs) - [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases) - [Benchmark Evaluations](/docs/benchmark-evaluations) {% /related-card-grid %} ## Source confidence Code-backed: the current output-mapping modal defines identifier and output-column selection, mapping preview, unmatched rows, metadata, and creation of an output-only Run. The Runs workspace keeps that imported evidence distinct from saved Harness Versions. --- id: troubleshooting.overlapping-rubrics title: Overlapping Rubrics summary: Resolve rubrics that ask the same question or create contradictory evidence. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/overlapping-rubrics --- # Overlapping Rubrics Use this when multiple rubrics appear to fail the same behavior or produce contradictory evidence. ## Symptom A single output triggers several rubric failures that all describe the same defect, or two rubrics disagree about whether the same behavior should pass. Review context becomes noisy because failures are counted more than once. ## Likely causes - A new rubric was created without checking existing approved rubrics. - One rubric tests several behaviors and overlaps narrower checks. - Policy revisions changed the behavior boundary but old rubrics remained active. - Applicability rules bind multiple rubrics to the same irrelevant cases. ## Diagnostic checks - Group failures by output and identify rubrics that cite the same evidence. - Compare rubric pass/fail wording against linked policies. - Check rubric versions and whether old versions are still bound to benchmark cases. - Inspect applicability rules for rubrics that should be mutually exclusive. ## Fix - Choose the Rubric definition that should govern future evidence and revise or version the affected governed objects through their available controls. Do not claim a merge or retire action unless it is visible in the current surface. - Split multi-behavior rubrics into one binary check each. - Revise applicability so rubrics apply only to the intended cases. - Rerun affected benchmark slices after approved rubric changes. ## Prevention - Search existing rubrics before approving a new one. - Require a passing and failing example for every rubric boundary. - Review overlap during policy/rubric approval. - Treat rubric edits as versioned changes that can affect benchmark comparisons. ## Related task pages {% related-card-grid title="Related task pages" %} - [Create a rubric](/docs/correctness-governance/policies-and-rubrics) - [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics) - [Read run results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Rubrics](/docs/object-model/rubrics) - [Rubric versions](/docs/governance/rubric-versioning) - [Applicability logic](/docs/object-model/applicability-logic) {% /related-card-grid %} ## Source confidence Code-backed: Rubric list and detail surfaces expose Rubric identity and versions; linked Policy Cases and evaluation results support comparison of scope and observed failures. The product does not expose a universal merge-or-retire workflow. --- id: troubleshooting.overly-broad-policies title: Overly Broad Policies summary: Narrow policies that are too vague to guide review, rubrics, or human reviews. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/overly-broad-policies --- # Overly Broad Policies Use this when a policy is too general to guide review, rubric writing, or human reviews. ## Symptom Reviewers agree with the policy in principle but cannot decide whether a specific output satisfies it. Rubrics derived from the policy become vague, noisy, or impossible to apply consistently. ## Likely causes - The policy describes a value such as “be helpful” without a behavioral boundary. - Several domain rules were combined into one policy. - The policy lacks linked cases that show pass and fail examples. - The policy was approved from one expert comment without broader review. ## Diagnostic checks - Ask whether the policy can produce a binary rubric without adding new assumptions. - Find linked cases that should pass and fail the policy. - Compare the policy to existing standards for duplicate or conflicting scope. - Review benchmark failures to see whether the policy explains the evidence. ## Fix - Narrow the policy to one behavior boundary. - Split unrelated requirements into separate policies. - Add linked cases and applicability before approval. - Use the Policy's own governed approval and version controls after revision. A Contribution Checkpoint confirms Contribution progress; it is not automatically Policy approval. ## Prevention - Draft policies from repeated judgments, not one-off comments. - Require applicability and examples before approval. - Review policy overlap during standards maintenance. - Version revisions so old benchmark evidence remains interpretable. ## Related task pages {% related-card-grid title="Related task pages" %} - [Create a policy](/docs/correctness-governance/policies-and-rubrics) - [Using expert judgment](/docs/concepts/correctness-elicitation) - [Resolve conflicting correctness evidence](/docs/governance/conflict-resolution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Policies](/docs/object-model/policies) - [Policy versions](/docs/governance/policy-versioning) - [Correctness Elicitation](/docs/concepts/correctness-elicitation) {% /related-card-grid %} ## Source confidence Code-backed: Policy list and detail routes expose governed Policy content, approval, versions, linked Cases, and Rubrics. This page keeps Contribution Checkpoints separate from Policy approval. --- id: troubleshooting.permissions title: Permissions troubleshooting summary: Separate Project membership from Expert Contribution assignment and readiness when an expert has no tasks available. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/permissions --- # Permissions troubleshooting Use this when project role, reviewer assignment, or approval ownership prevents someone from completing the intended work. ## Symptom A user can access Teammately but cannot open the expected Project, or an expert reports **no tasks available** after opening their Contribution link. These symptoms belong to different owning surfaces. ## Likely causes - The user or group is not listed under **Project Settings → Project Members**. - The Contribution was requested for a different recipient. - The Contribution exists but its next Task or Checkpoint is not ready. - The current IP is blocked before Project membership is evaluated. ## Diagnostic checks - Confirm that the user or group appears in **Project Members** for the intended Project. - Open the Contribution from the sender-side Expert Contributions workspace and verify its recipient and status. - Check whether the expert has an executable Task or Checkpoint, not merely whether the Contribution exists. - If **Access Restricted** is visible, resolve the IP allowlist first. ## Fix - Add the correct user or group under **Project Settings → Project Members** when Project access is missing. - Correct the Contribution recipient through the owning Contribution workflow when the request went to the wrong person. - Resolve lifecycle or readiness problems in Expert Contributions when access succeeds but no Task can be entered. - Do not infer a permission from a role label or edit governed artifacts merely to make a control appear. ## Prevention - Separate reviewer access from approval authority in review setup. - Check assignments before launching a review session. - Use a small pilot Contribution before assigning a larger specialist cohort. - Keep project membership changes visible to review owners. ## Related task pages {% related-card-grid title="Related task pages" %} - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) - [Reviewer assignments and statuses](/docs/expert-contributions) - [Using checkpoints](/docs/expert-contributions/complete-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Permissions](/docs/reference/permissions) - [Roles and permissions](/docs/governance/roles-and-permissions) - [Expert Contributions](/docs/expert-contributions) {% /related-card-grid %} ## Source confidence Code-backed: Project Members and the redirected Project Permissions route define current Project access management; Contribution status and runtime navigation distinguish access from executable-task readiness. The Access Restricted route defines the separate IP boundary. --- id: troubleshooting.stale-dimensions title: Stale Dimensions summary: Refresh Dimensions and ontology values that no longer explain the current behavior space without rewriting historical evidence. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/stale-dimensions --- # Stale Dimensions ## Symptoms - New Cases repeatedly fall into **other**, unknown, or no value. - An ontology value refers to a product state or source hierarchy that no longer exists. - Important failures concentrate in metadata that no Dimension represents. - A coverage plan looks balanced under old labels but reviewers describe a new boundary. - Two values have become indistinguishable after a product change. ## Likely causes - Product behavior or source authority changed while the coverage vocabulary did not. - New Cases reveal an axis the existing schema never represented. - Ontology values were renamed or repurposed without reviewing old classifications. - The apparent staleness is actually incomplete Case classification. ## Confirm staleness First distinguish a stale schema from incomplete classification. Sample new and old Cases using the current Dimension definition. If the existing values still describe the behavior and only new Cases are unlabeled, repair classification. If reviewers need a new concept, different source authority, or changed applicability to classify consistently, the Dimension or ontology may be stale. ## Fix 1. Record the change that made the current vocabulary inadequate. 2. Inspect the Dimension definition, values, examples, origin, and where it is used in coverage plans. 3. Decide whether to rename a value, add a value, split the Dimension, replace it, or preserve it with a historical time boundary. 4. Review representative Cases against the proposed schema before broad reclassification. 5. Reclassify affected current Cases and inspect whether coverage gaps or target distributions changed. 6. Update **Coverage Management → Get Started** and the overview deliberately. Do not change selected Dataset membership merely to preserve an old-looking distribution. 7. Create a new Snapshot when the classification or selected evidence boundary used by the Benchmark changes. Historical Snapshots and Runs should retain their original interpretation. A new Dimension schema can supersede the current planning model without making the old model disappear. ## Prevention Review Dimension definitions alongside product and source changes, keep representative examples for each ontology value, and inspect unclassified or catch-all Cases regularly. Name the schema and Snapshot boundary used when segment results inform a decision. {% example-demo title="Example: channel labels stop explaining escalation risk" %} A support Benchmark classifies Cases only by email and chat. After voice transcripts arrive, experts find that synchronous versus asynchronous interaction—not channel name—explains escalation behavior. The team creates a clearer interaction-mode Dimension, samples old and new Cases, updates Coverage Management, and creates a new Snapshot. Older results remain labeled under the prior schema. {% /example-demo %} ## Source confidence Code-backed: Dimension types, Dimensions and Ontology, Coverage Management, and Get Started show the editable vocabulary and its use in benchmark planning. The product does not automatically prove conceptual staleness; the trigger comes from changed evidence and reviewer interpretation. ## Related task pages {% related-card-grid title="Related workflows" %} - [Refresh coverage after product change](/docs/coverage-engineering/coverage-refresh) - [Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) - [Detect and route stale evidence](/docs/governance/staleness-detection) - [Dimension classification troubleshooting](/docs/troubleshooting/dimension-classification) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Ontology](/docs/object-model/ontology) - [Versions, staleness, and resolution](/docs/object-model/versions-staleness-and-resolution) {% /related-card-grid %} --- id: troubleshooting.unbalanced-coverage title: Unbalanced Coverage summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/unbalanced-coverage --- # Unbalanced Coverage Use this when benchmark evidence overrepresents some behavior and misses other behavior that matters before human review. ## Symptom A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent. ## Likely causes - Case import mirrored production volume instead of product risk. - Dimensions or metadata do not represent the behavior areas that matter. - Synthetic cases were added in one area but not reviewed across the full risk boundary. - Coverage was not refreshed after new signals, policy changes, or failures. ## Diagnostic checks - Inspect **Benchmark Datasets → Representation** and the Coverage Management overview by the configured Coverage Facets. - Look for policies or rubrics with very few applicable cases. - Compare production or expert signals with benchmark case distribution. - Inspect whether new cases are candidate-only or actually in use. ## Fix - Add representative, boundary, or adversarial cases for undercovered areas. - Refresh dimensions or metadata before judging balance. - Review and promote candidate cases into the benchmark only after context is complete. - Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance. ## Prevention - Define coverage targets before benchmark runs. - Track candidate and in-use cases separately. - Review coverage after policy, rubric, or product behavior changes. - Do not use aggregate scores without a coverage note. ## Related task pages {% related-card-grid title="Related task pages" %} - [Inspect Dataset representation](/docs/benchmark-datasets/representation) - [Configure Coverage Management](/docs/coverage-management/get-started) - [Work with Coverage Stories](/docs/coverage-management/coverage-stories) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Case pool](/docs/object-model/case-pool) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %} ## Source confidence Code-backed: Benchmark Dataset Representation, Coverage Management, Get Started, and Coverage Stories expose the current selected distribution, benchmark denominator, guidance, and named gaps used for this diagnosis. --- id: troubleshooting.unclear-cases title: Unclear Cases summary: Repair Cases that informed reviewers cannot interpret consistently from the supplied input, context, and output identity. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/unclear-cases --- # Unclear Cases ## Symptoms - Reviewers ask what they are supposed to judge. - The same Case produces incompatible readings because a source, prior turn, time boundary, or user state is absent. - A candidate answer is visible, but it is unclear whether it is the judged output or reference behavior. - The Case title or metadata implies context that the review presentation does not contain. - Experts can answer only by inventing assumptions. ## Likely causes - A prior conversation turn, source, time boundary, or user state is missing. - Candidate and reference outputs are swapped or unlabeled. - One Case combines multiple situations or requested judgments. - Important instructions live in metadata or private project knowledge instead of visible context. ## Diagnose the Case, not the reviewer 1. Open the Case in its normal review presentation and read only what the assigned expert can see. 2. Identify the judged input and output separately. For conversations, confirm the turn order and which response is under review. 3. List every fact required to make the judgment, then verify each fact is present as Case content, context, source material, or an explicitly linked standard. 4. Check whether metadata is being used as hidden instruction rather than visible context. 5. Compare reviewer rationale. Repeated invented assumptions usually reveal the missing boundary. ## Fix - Add or map the missing context when it belongs to the real situation. - Correct the output attachment when candidate and reference behavior were swapped. - Split a Case that combines two independently judged situations. - Narrow the question or applicability boundary when multiple interpretations are legitimate. - Mark the Case unresolved or keep it as candidate material when authoritative evidence is unavailable. - Create a new Case version when the reviewable content changes; do not silently rewrite evidence used by an existing Snapshot. After correction, have a reviewer who did not author the Case explain the situation and requested judgment. If they must rely on private project knowledge, the Case is still incomplete. ## Prevention Review the Case through the same presentation an expert will see, label output identity explicitly, and include every fact required for judgment. Sample imported and generated Cases before Benchmark selection, and version content changes after evidence exists. {% example-demo title="Example: recommendation without the user's equipment" %} A Case asks whether a product is compatible but omits the equipment model mentioned in an earlier conversation turn. Reviewers alternate between pass and fail. The operator restores the prior turn, confirms the candidate output identity, versions the Case, and repeats the affected review. The old Snapshot retains the earlier ambiguous version. {% /example-demo %} ## Source confidence Code-backed: Cases, Case review, Contribution preview, and Case-content schemas establish the separate input, contextual-material, and presentation boundaries used here. Whether the supplied facts are sufficient for a domain judgment remains reviewer-owned. ## Related task pages {% related-card-grid title="Related workflows" %} - [Case model overview](/docs/assets) - [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Case Versioning](/docs/governance/case-versioning) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Cases](/docs/object-model/cases) - [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases) - [Metadata and context](/docs/reference/metadata-and-context) {% /related-card-grid %} --- id: troubleshooting.unrealistic-synthetic-cases title: Synthetic Cases That Feel Unrealistic summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use. kind: error product_area: troubleshooting status: stable updated: 2026-09-07 canonical: /docs/troubleshooting/unrealistic-synthetic-cases --- # Synthetic Cases That Feel Unrealistic ## Symptoms - Generated language sounds artificial or describes a user who would not exist in the product. - Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere. - The Case combines incompatible ontology values or violates product constraints. - Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark. - Many generated candidates repeat the same template with superficial wording changes. ## Likely causes - The synthesis request lacks representative source Cases or product constraints. - A coverage tuple combines values that cannot coexist in the target system. - A Comparison Direction is too broad, stale, or duplicative. - Generation is being judged as finished benchmark evidence instead of candidate material. ## Diagnose the generation boundary 1. Identify the entry surface. **Assets → Cases → Synthesize** creates reusable project Case candidates; **Coverage Management → Case Foundry** coordinates candidates for a named benchmark need. 2. Compare the generated Case with real in-use Cases from the same behavior region. 3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires different inputs or directions. 4. Check whether the generator lacked constraints that operators hold only in private memory. 5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not selected Benchmark Dataset evidence. ## Fix - Add representative source Cases or explicit product constraints to the synthesis context. - Narrow, edit, archive, or delete a Comparison Direction that produces implausible combinations. - Generate from a specific missing coverage tuple rather than asking for generic diversity. - Edit a promising candidate when its behavioral boundary is sound and only presentation is weak. - Reject candidates whose premise is impossible or whose source context cannot be justified. - Review duplicates before adding any generated Case to a Benchmark. Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use. ## Prevention Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Keep Comparison Directions narrow, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot. {% example-demo title="Example: impossible refund request" %} Synthesis produces a Case where an anonymous visitor requests an enterprise-contract refund and supplies internal approval metadata in chat. The tuple is valuable—exception request with conflicting authority—but the realization is impossible. The team adds authentication and channel constraints, narrows the Comparison Direction, and regenerates. The realistic candidate still goes through Case review before selection. {% /example-demo %} ## Source confidence Code-backed: the Assets Case synthesizer, generated Case cards, synthesis lifecycle, and benchmark-scoped Case Review establish the candidate-generation and admission boundary. Case Foundry provides the benchmark-scoped coordination path. Realism remains domain-specific and requires customer product knowledge. ## Related task pages {% related-card-grid title="Related workflows" %} - [Synthesize Cases](/docs/coverage-engineering/synthesize-cases) - [Comparison Directions](/docs/assets/comparison-directions) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Case Pool](/docs/coverage-engineering/case-pool) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Case Pool](/docs/object-model/case-pool) - [Cases](/docs/object-model/cases) - [Coverage dimensions](/docs/object-model/coverage-dimensions) {% /related-card-grid %} --- id: troubleshooting.weak-applicability-logic title: Weak Applicability Logic summary: Fix standards that are applied to the wrong cases or skipped where they matter. kind: error product_area: troubleshooting status: stable updated: 2026-08-23 canonical: /docs/troubleshooting/weak-applicability-logic --- # Weak Applicability Logic Use this when policies or rubrics fire on irrelevant cases or miss cases where they should apply. ## Symptom Benchmark results show failures that reviewers consider irrelevant, or important cases skip the standards that should govern them. The issue appears as false positives, false negatives, or confusing policy-level result counts. ## Likely causes - The applicability condition uses a broad keyword or metadata field that does not prove the behavior is in scope. - Cases lack the metadata or source context the applicability rule depends on. - A policy boundary changed but applicability was not revised. - Conversation context or retrieved-source state is not represented in the case. ## Diagnostic checks - Inspect included and excluded cases side by side. - Identify the exact source signal the applicability rule depends on. - Check whether case metadata, dimensions, or context fields are missing or stale. - Review recent policy or rubric revisions for boundary changes. ## Fix - Rewrite applicability around source-backed signals, not broad topic labels. - Add missing metadata or context before relying on the rule. - Create examples that should be included and excluded, then test the boundary. - Approve and version the corrected governed objects, then create the Benchmark boundary and Runs needed to evaluate the revised applicability. ## Prevention - Define applicability before benchmark runs, not after reading failures. - Keep case metadata and dimensions current. - Review applicability whenever a policy or rubric version changes. - Keep included and excluded example Cases linked to the governed standard where the current surface supports them. ## Related task pages {% related-card-grid title="Related task pages" %} - [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics) - [Work with Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Applicability logic](/docs/object-model/applicability-logic) - [Policies](/docs/object-model/policies) - [Coverage dimensions](/docs/object-model/coverage-dimensions) {% /related-card-grid %} ## Source confidence Code-backed: Policy and Rubric detail routes expose governed scope and linked Cases, while Benchmark Evaluation results expose which evaluator outcomes appeared for Cases. The boundary-testing method remains a human interpretation of those inspectable artifacts.