# Unexpected result changes
Generated: 2026-11-13T08:01:24.845Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: workflow.troubleshooting.result-changes
title: Unexpected result changes
summary: A later result differs from the baseline and the cause is unclear.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-11
canonical: /docs/troubleshooting/result-changes
---
# Unexpected result changes
## Symptom
A later result differs from the baseline and the cause is unclear.
## Likely causes
| What you observe | Focused guide |
| --- | --- |
| Candidate or assessment conditions differ | [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) |
| Execution completeness differs | [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs) |
| The assessed criteria differ | [Weak Applicability Logic](/docs/troubleshooting/weak-applicability-logic) |
## Fix
Compare the actual candidate identity, Benchmark Version, Case population, applicable criteria, scoring settings, trial selection, and judgment corrections. Trace meaningful movement into individual Cases before attributing it to an implementation change.
Follow the focused guide for the actual state you observe; keep accepted records and explicit human decisions intact.
## Prevention
Preserve exact object and version identities, review the relevant source context, and inspect the accepted state before comparing outcomes.
## Related task pages
{% related-card-grid title="Return to the workflow" %}
- [Complete your first alignment loop](/docs/quickstart)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Diagnose the observed state" %}
- [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs)
- [Weak Applicability Logic](/docs/troubleshooting/weak-applicability-logic)
{% /related-card-grid %}
---
id: troubleshooting.benchmark-results-changed
title: Benchmark Results Changed Unexpectedly
summary: Diagnose result changes across target behavior, benchmark cases, standards, and versions.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/benchmark-results-changed-unexpectedly
---
# Benchmark Results Changed Unexpectedly
Use this when benchmark results change and the team is not sure whether the cause is model behavior or an artifact change.
## Symptom
A pass rate, policy-level result, rubric result, or case-level outcome changes between runs even though the expected candidate behavior did not obviously change.
## Likely causes
- Benchmark membership changed through case import, promotion, removal, or refresh.
- Policy, rubric, or applicability versions changed between runs.
- A different saved Harness Version, execution setting, or imported output-only Run was inspected.
- Run metadata or benchmark version selection differs from the previous run.
## Diagnostic checks
- Compare benchmark version, case count, policy versions, rubric versions, and applicability boundary.
- Inspect changed Cases and confirm whether each result came from a managed Run or imported output-only Run.
- Check run metadata for candidate, prompt, retrieval, or model differences.
- Open policy/rubric result changes and trace them to exact cases.
## Fix
- If artifacts changed, label the comparison as an artifact-boundary change rather than a pure behavior regression.
- If the candidate identity or settings changed, run the intended saved Harness Version with the intended settings.
- If the Benchmark boundary changed, create and name the appropriate Snapshot and Benchmark Version; do not rewrite the older Run.
- If the cause remains unclear, hold downstream action until the changed inputs, criteria, or results can be explained.
## Prevention
- Record benchmark, case, policy, rubric, and candidate versions for every run.
- Use comparison views before summarizing score movement.
- Treat mapping, coverage, and standard changes as review-context boundaries.
- Keep previous runs reproducible for audit.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
- [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Versions, staleness, and resolution](/docs/object-model/versions-staleness-and-resolution)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Benchmark Versioning](/docs/governance/benchmark-versioning)
{% /related-card-grid %}
---
id: troubleshooting.benchmark-runs
title: Benchmark run troubleshooting
summary: Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/benchmark-runs
---
# Benchmark run troubleshooting
## Symptoms
- The evaluation surface has no Benchmark Version to run.
- Output import or mapping cannot identify a Case or output column.
- A Run is created but remains empty, incomplete, or failed.
- Results appear under the wrong candidate label or Harness Version.
- Two Runs show a score change but do not share a comparable snapshot boundary.
## Likely causes
- No immutable Snapshot exists for the intended evaluation set.
- Imported outputs are mapped to the wrong Case or column.
- The saved Harness Version or candidate metadata does not match the evaluated system.
- The Run is partial, failed, or being compared across different Benchmark Versions.
## Check the boundary before the failure
1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version.
2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run.
3. For managed execution, confirm the saved Harness Version and recorded settings. For imported results, confirm that the external candidate context describes the system that produced the submitted responses.
4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior.
5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates.
## Fix
- **No runnable version:** finish Case selection and create a Snapshot first.
- **Missing imported outputs:** append only missing Cases to an uncompleted Trial using the exact manifest IDs. Correcting an accepted response or adding Cases after completion requires a new Trial in the external Run or a new Run. Preserve the original results.
- **Wrong Harness or metadata:** create a correctly configured Run. Do not relabel completed results to represent a different system.
- **Partial managed failure:** inspect and preserve the existing Run's recorded outcomes. For another independent execution under unchanged conditions, use **Run again**. **Run evaluation** reopens matching work, including failed work. Publish a new Harness Version when its code changes, or create a new Snapshot and Benchmark Version when Case content or criteria change.
- **Confusing comparison:** compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed.
## Prevention
Create a Snapshot before evaluation. For managed execution, save the exact Harness Version. For imported results, preserve external candidate context and preview the mapping on representative Cases. Before comparing, confirm that every difference between the two Runs is intentional and visible.
## Recovery check
Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping.
{% example-demo title="Example: scores drop after output import" %}
A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Outputs](/docs/object-model/outputs)
{% /related-card-grid %}
---
id: troubleshooting.weak-applicability-logic
title: Weak Applicability Logic
summary: Fix standards that are applied to the wrong cases or skipped where they matter.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/weak-applicability-logic
---
# Weak Applicability Logic
Use this when policies or rubrics fire on irrelevant cases or miss cases where they should apply.
## Symptom
Benchmark results show failures that reviewers consider irrelevant, or important cases skip the standards that should govern them. The issue appears as false positives, false negatives, or confusing policy-level result counts.
## Likely causes
- The applicability condition uses a broad keyword or metadata field that does not prove the behavior is in scope.
- Cases lack the metadata or source context the applicability rule depends on.
- A policy boundary changed but applicability was not revised.
- Conversation context or retrieved-source state is not represented in the case.
## Diagnostic checks
- Inspect included and excluded cases side by side.
- Identify the exact source signal the applicability rule depends on.
- Check whether case metadata, dimensions, or context fields are missing or stale.
- Review recent policy or rubric revisions for boundary changes.
## Fix
- Rewrite applicability around source-backed signals, not broad topic labels.
- Add missing metadata or context before relying on the rule.
- Create examples that should be included and excluded, then test the boundary.
- Approve and version the corrected governed objects, then create the Benchmark boundary and Runs needed to evaluate the revised applicability.
## Prevention
- Define applicability before benchmark runs, not after reading failures.
- Keep case metadata and dimensions current.
- Review applicability whenever a policy or rubric version changes.
- Keep included and excluded example Cases linked to the governed standard where the current surface supports them.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Build policies and rubrics](/docs/operating-manual/build-policies-and-rubrics)
- [Policies and Rubrics](/docs/correctness-governance/policies-and-rubrics)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Applicability logic](/docs/object-model/applicability-logic)
- [Policies](/docs/object-model/policies)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
{% /related-card-grid %}