{"query":"Benchmark Results Changed Unexpectedly","corpusVersion":"local","generatedAt":"2026-09-13T04:41:28.337Z","results":[{"blockId":"troubleshooting.benchmark-results-changed#benchmark-results-changed-unexpectedly","pageId":"troubleshooting.benchmark-results-changed","title":"Benchmark Results Changed Unexpectedly","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#benchmark-results-changed-unexpectedly","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"benchmark-results-changed-unexpectedly","kind":"error","productArea":"troubleshooting","score":2966.314059461326,"reasons":["search_match","title_match","term_match","prefix_or_fuzzy_match"],"markdown":"# Benchmark Results Changed Unexpectedly\n\nUse this when benchmark results change and the team is not sure whether the cause is model behavior or an artifact change."},{"blockId":"troubleshooting.benchmark-results-changed#symptom","pageId":"troubleshooting.benchmark-results-changed","title":"Symptom","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#symptom","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"symptom","kind":"error","productArea":"troubleshooting","score":1359.2347678310957,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Symptom\n\nA pass rate, policy-level result, rubric result, or case-level outcome changes between runs even though the expected candidate behavior did not obviously change."},{"blockId":"troubleshooting.benchmark-results-changed#prevention","pageId":"troubleshooting.benchmark-results-changed","title":"Prevention","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#prevention","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"prevention","kind":"error","productArea":"troubleshooting","score":1348.9272799188125,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Prevention\n\n- Record benchmark, case, policy, rubric, and candidate versions for every run.\n- Use comparison views before summarizing score movement.\n- Treat mapping, coverage, and standard changes as review-context boundaries.\n- Keep previous runs reproducible for audit."},{"blockId":"troubleshooting.benchmark-results-changed#fix","pageId":"troubleshooting.benchmark-results-changed","title":"Fix","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#fix","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"fix","kind":"error","productArea":"troubleshooting","score":1131.2751552307152,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Fix\n\n- If artifacts changed, label the comparison as an artifact-boundary change rather than a pure behavior regression.\n- If the candidate identity or settings changed, run the intended saved Harness Version with the intended settings.\n- If the Benchmark boundary changed, create and name the appropriate Snapshot and Benchmark Version; do not rewrite the older Run.\n- If the cause remains unclear, hold downstream action until the changed evidence can be explained."},{"blockId":"troubleshooting.benchmark-results-changed#diagnostic-checks","pageId":"troubleshooting.benchmark-results-changed","title":"Diagnostic checks","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#diagnostic-checks","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"diagnostic-checks","kind":"error","productArea":"troubleshooting","score":1129.1884284306216,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Diagnostic checks\n\n- Compare benchmark version, case count, policy versions, rubric versions, and applicability boundary.\n- Inspect changed Cases and confirm whether each result came from a managed Run or imported output-only Run.\n- Check run metadata for candidate, prompt, retrieval, or model differences.\n- Open policy/rubric result changes and trace them to exact cases."},{"blockId":"troubleshooting.benchmark-results-changed#likely-causes","pageId":"troubleshooting.benchmark-results-changed","title":"Likely causes","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#likely-causes","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"likely-causes","kind":"error","productArea":"troubleshooting","score":1122.2892866115556,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Likely causes\n\n- Benchmark membership changed through case import, promotion, removal, or refresh.\n- Policy, rubric, or applicability versions changed between runs.\n- A different saved Harness Version, execution setting, or imported output-only Run was inspected.\n- Run metadata or benchmark version selection differs from the previous run."},{"blockId":"troubleshooting.benchmark-results-changed#related-task-pages","pageId":"troubleshooting.benchmark-results-changed","title":"Related task pages","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#related-task-pages","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"related-task-pages","kind":"error","productArea":"troubleshooting","score":1121.3750070875362,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Related task pages\n\n{% related-card-grid title=\"Related task pages\" %}\n- [Compare Harness Versions](/docs/benchmark-evaluations/compare)\n- [Read run results](/docs/benchmark-evaluations/inspect-results)\n- [Inspect execution settings](/docs/benchmark-evaluations/execution-settings)\n{% /related-card-grid %}"},{"blockId":"troubleshooting.benchmark-results-changed#related-reference-pages","pageId":"troubleshooting.benchmark-results-changed","title":"Related reference pages","pageTitle":"Benchmark Results Changed Unexpectedly","url":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","humanUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly#related-reference-pages","markdownUrl":"https://teammately.ai/docs/troubleshooting/benchmark-results-changed-unexpectedly.md","sectionId":"related-reference-pages","kind":"error","productArea":"troubleshooting","score":1095.1327375200588,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Related reference pages\n\n{% related-card-grid title=\"Related reference pages\" %}\n- [Versions, staleness, and resolution](/docs/object-model/versions-staleness-and-resolution)\n- [Benchmark Evaluations](/docs/benchmark-evaluations)\n- [Benchmark versioning](/docs/governance/benchmark-versioning)\n{% /related-card-grid %}"}]}