# Refreshing a Benchmark from New Signals Generated: 2026-09-13T04:43:12.846Z Source build: local Canonical docs: https://teammately.ai/docs --- id: playbooks.benchmark-refresh title: Refreshing a Benchmark from New Signals summary: Update benchmark coverage and standards when new cases, review findings, or product changes appear. kind: recipe product_area: playbooks status: stable updated: 2026-08-23 canonical: /docs/playbooks/refreshing-a-benchmark-from-new-signals --- # Refreshing a Benchmark from New Signals Use this playbook when customer-provided production examples, Contribution findings, source changes, or target-system changes show that the current Benchmark no longer represents the intended behavior. ## Classify the signal before editing Identify whether the signal changes the correctness standard, coverage vocabulary, available Cases, selected Dataset membership, or candidate behavior. These changes have different owners and version consequences. ## Refresh path 1. Record the signal, source, effective date, and latest Benchmark Version it may affect. 2. Route a source or rule change to Reference Materials, Expert Contributions, and Correctness Governance. 3. Route a changed behavior axis to Coverage Facets and review the impact on existing classifications. 4. Route missing examples to Assets Cases, Synthesis, or Coverage Management Case Foundry; review candidates in Case Review. 5. Change selected membership under Benchmark Datasets and create a new Dataset Snapshot. 6. Create the Benchmark Version that represents the revised Dataset and governed evaluator boundary. 7. Run the same saved Harness Version against the new boundary when you need to isolate the benchmark change. Run a new Harness Version separately when candidate behavior also changed. 8. Explain results using the two named boundaries; Compare does not implicitly normalize evidence across different Benchmark Versions. ## Preserve history Do not edit an older Snapshot or Run to resemble current reality. Staleness means the evidence may no longer answer the current question; it does not erase what the old version measured. {% example-demo title="Refund policy change" %} A support team adds a new outage-credit exception. The existing Benchmark remains valid for the former rule but has no Cases for the new exception. Experts approve revised applicability and Rubrics; Case Foundry prepares eligible, ineligible, and missing-account-evidence Cases; Case Review admits them into the current Dataset. A new Snapshot and Benchmark Version show that the unchanged Harness still passes ordinary refunds but fails the new outage boundary. {% /example-demo %} ## Evidence to collect - The new signal, its source, and the benchmark version it affects. - Cases, reference responses where supported, Policies, Rubrics, or Coverage Facets changed by the signal. - Human approval for whether the change updates standards, coverage, or both. - Previous and refreshed Snapshot and Benchmark Version IDs, saved Harness Version, settings, and Run Metadata. - Case-level evidence that explains movement caused by the revised boundary rather than candidate behavior. ## Related docs {% related-card-grid title="Related docs" %} - [Coverage Refresh](/docs/coverage-engineering/coverage-refresh) - [Staleness Detection](/docs/governance/staleness-detection) - [Work with Dataset Snapshots](/docs/benchmark-datasets/snapshots) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Read run results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Source confidence Doctrine-backed: the approved lifecycle routes new signals to the artifact that owns the change and preserves historical evidence. Linked code-backed pages define the current Coverage, Dataset, Snapshot, governance, and Evaluation operations. --- id: coverage.refresh title: Refresh coverage after product change summary: Reconcile coverage facets, Cases, benchmark membership, and Snapshots after the target system or its evidence changes. kind: task product_area: coverage_engineering status: stable updated: 2026-09-07 canonical: /docs/coverage-engineering/coverage-refresh --- # Refresh coverage after product change ## When to use it Refresh coverage when new Cases reveal an unrepresented behavior, source material or product behavior changes, experts qualify an earlier assumption, or a governed Policy changes which situations matter. This is a coordinated workflow across Coverage Engineering—not a single refresh action. ## Prerequisites - Name the changed signal and the date or version at which it changed. - Identify the Benchmark whose claims may be affected. - Preserve the current Snapshot and historical Runs; do not edit them to resemble the new state. - Decide who can confirm the changed behavior and who owns the resulting Benchmark Version. ### Task steps: Refresh benchmark coverage 1. Open the Benchmark's **Coverage Management** overview and identify which coverage claim is no longer supported. 2. Review **Coverage Facets**. Update Dimensions, ontology values, Project Topics, or Case Construction Patterns only when the behavior model itself changed. 3. Return to the **Case Pool**. Source, upload, draft, or synthesize candidate Cases for the missing or changed region. 4. Inspect the candidates for source context, realistic inputs, duplication, and the intended coverage labels. Keep uncertain Cases out of benchmark use. 5. Use **Coverage Management → Get Started** and the overview to update coverage guidance. Use **Case Review** and **Benchmark Datasets** to change selected Cases deliberately. 6. Create a new Dataset Snapshot and Benchmark Version for the revised evidence boundary. 7. Run a new evaluation when current candidate evidence is required. Compare it with older Runs using the named Benchmark Versions. ## Object and state changes A refresh may change coverage-facet definitions, Case classifications, candidate Cases, selected benchmark Cases, and the next Snapshot. It does not rewrite an earlier Snapshot or make its Runs invalid. Older results remain evidence for their original version; the new version answers the current coverage question. If only candidate behavior changed, keep the Benchmark Version fixed and run the new candidate against it. If the Case set, applicable standards, or coverage boundary changed, create a new Benchmark Version before interpreting a new Run as comparable. ## Success criteria - The changed product reality maps to an explicit coverage facet or documented boundary. - Candidate Cases have enough source context to be reviewed and are not mistaken for in-use benchmark evidence. - The new selected set addresses the gap without silently removing still-important behavior. - The new Snapshot names a reproducible evidence boundary. - Comparisons distinguish candidate changes from Benchmark Version changes. ## Common failure modes - Treating refresh as a single button and missing a changed facet, Case set, or Snapshot boundary. - Rewriting a historical Snapshot instead of creating a new one. - Adding generated or newly sourced Cases to a Benchmark before review. - Comparing Runs without naming whether the candidate, Benchmark Version, or both changed. {% example-demo title="A newly supported exception" %} A support assistant gains an approved exception path for one account tier. The team adds or revises the account-tier ontology, sources Cases for eligible and ineligible requests, reviews them in Case Review, updates the coverage guidance, and changes the selected Benchmark Dataset. A new Snapshot freezes the revised membership. Previous Runs still describe the old rule; new Runs evaluate the approved exception boundary. {% /example-demo %} ## Source confidence Code-backed: Coverage Management, Coverage Facets, the Case Pool, Case Review, Benchmark Datasets, and Dataset Snapshots establish the current sequence and the objects that can change. The decision that a product change requires a refresh remains a team-owned interpretation of evidence. ## Related reference pages {% related-card-grid title="Continue the workflow" %} - [Coverage gaps](/docs/coverage-engineering/coverage-gaps) - [Dimensions and ontology](/docs/coverage-engineering/dimensions-ontology) - [Case Pool](/docs/coverage-engineering/case-pool) - [Plan benchmark coverage](/docs/coverage-engineering/plan-benchmark-coverage) - [Benchmark Snapshots](/docs/coverage-engineering/benchmark-snapshots) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Diagnose refresh problems" %} - [Stale Dimensions](/docs/troubleshooting/stale-dimensions) - [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage) - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) {% /related-card-grid %} --- id: governance.staleness-detection title: Detect and route stale correctness evidence summary: Identify which current claims need review after Cases, standards, coverage, sources, or target behavior change. kind: task product_area: governance status: stable updated: 2026-09-07 canonical: /docs/governance/staleness-detection --- # Detect and route stale correctness evidence ## What staleness means Staleness means a downstream claim may no longer be supported by the current upstream state. It is a review signal, not automatic deletion and not a claim that historical evidence was invalid when produced. Teammately objects change independently. A revised Policy may affect linked Rubrics and current benchmark interpretation without changing the exact evidence captured in an earlier Snapshot. A changed Harness may require a new Run while leaving the Benchmark Version unchanged. ## Common triggers - A Case input, context, source attachment, or supported reference output changes. - A Policy rule, scope, approval, or applicable boundary changes. - A Rubric criterion or link changes. - Coverage facets or selected Case membership change. - A source becomes superseded or the target product changes. - A saved Harness version changes before candidate comparison. ## Prerequisites - The changed object and its earlier and current versions can be identified. - The team can trace current downstream artifacts that rely on the changed fact. - An owner can decide whether current evidence needs qualification, replacement, or no action. ### Task steps: Assess and route staleness 1. Name the changed object, its earlier and current versions, and the reason for change. 2. Identify downstream artifacts that rely on the changed fact: linked standards, selected Cases, Snapshots, Runs, comparisons, or customer-owned human review context. 3. Classify each artifact as historically valid, current and unaffected, current but requiring qualification, or requiring replacement. 4. Route the correction to the owning workflow: edit a Case, govern a new Policy or Rubric version, refresh coverage, create a Snapshot, or run a saved Harness again. 5. Preserve the old version and its evidence. Add a note that states which boundary the evidence still supports. 6. Confirm that current navigation and handoff material point to the new canonical version. ## Object and state changes A changed object does not make every connected artifact unusable. If a Policy wording change does not affect a particular Case or Rubric, document that determination. If a candidate configuration changes, create a new Run rather than a new Benchmark Version. If selected membership changes, create a new Snapshot rather than editing an old one. Comparison Directions have a narrower advisory state. **Potentially stale** applies to untouched AI-suggested directions considered inconsistent with newer project context. It does not automatically apply to user-created or user-edited directions, and dismissing the label does not delete the direction or govern any downstream artifact. ## Success criteria - The trigger and affected version boundaries are named. - Historical evidence remains interpretable under its original boundary. - Current artifacts are explicitly unaffected, qualified, or routed to the owning workflow. - New Snapshots or Runs are created only when their respective evidence boundary changed. ## Common failure modes - Treating every connected artifact as invalid after one upstream change. - Editing a historical Snapshot or Run to resemble current state. - Creating a new Benchmark Version when only the Harness changed. - Confusing the advisory Comparison Direction label with governed-object staleness. {% example-demo title="Source document superseded" %} A new service policy supersedes the source used by twelve refund Cases. The operator preserves the old Snapshot and its Runs, updates affected current Cases, checks the linked Policy and Rubrics, and creates a new Snapshot. Two Cases describe historical behavior and remain unchanged with an explicit time boundary; ten move to the current version. {% /example-demo %} ## Source confidence Code-backed: Policy and Rubric types, Benchmark Version and Snapshot surfaces, and Run detail preserve the version boundaries needed for this assessment. Comparison Directions explicitly expose their narrower advisory stale label. Cross-object dependency assessment and the decision to rerun or revise remain owner-reviewed work. ## Related reference pages {% related-card-grid title="Related workflows" %} - [Versioning and Staleness](/docs/governance/versioning-and-staleness) - [Coverage Refresh](/docs/coverage-engineering/coverage-refresh) - [Benchmark Versioning](/docs/governance/benchmark-versioning) - [Comparison Directions](/docs/assets/comparison-directions) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Diagnose stale evidence" %} - [Stale Dimensions](/docs/troubleshooting/stale-dimensions) - [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) - [Weak applicability logic](/docs/troubleshooting/weak-applicability-logic) {% /related-card-grid %} --- id: governance.versioning-staleness title: Versioning and Staleness summary: Know when correctness objects changed and when old evidence may need review. kind: concept product_area: governance status: stable updated: 2026-09-07 canonical: /docs/governance/versioning-and-staleness --- # Versioning and Staleness ## Definition Versioning preserves the exact artifact state used by an earlier decision or evaluation. Staleness is the signal that current evidence, configuration, or interpretation may no longer support the same claim after a related object changes. A stale signal routes review; it does not automatically delete an artifact, invalidate every historical result, or approve a replacement. ## Why it matters Cases, Policies, Rubrics, Benchmark Versions, Harness Versions, and Improvement Session goals can change independently. Named versions keep old evidence interpretable. Staleness helps teams decide which current Datasets, evaluator links, Runs, or customer-owned human review context need attention before being treated as current. ## Where it appears in the product Use the owning object page to inspect its current version and activity. Use Dataset Snapshots and Benchmark Versioning for immutable evaluation boundaries. Use Staleness Detection to identify downstream artifacts affected by a change. Use Conflict Resolution when expert or source evidence disagrees about what the new governed state should be. ## Artifacts it affects Common triggers include changed Case input or materials, revised Policy scope, revised Rubric criteria, changed dataset membership, changed Harness configuration, and superseded source material. The responsible next action depends on the owner: correct a Case, approve a new standard version, refresh coverage, create a new Snapshot, run a new evaluation, or preserve an old result as historical context. Comparison Directions use a narrower stale signal. **Potentially stale** is an advisory label for untouched AI-suggested directions, not a versioned approval state and not automatic removal. User-created and user-edited directions remain user-owned even when Teammately considers them while avoiding duplicate suggestions. ## Operational check Name the changed artifact and version, identify which downstream claim depended on it, and decide whether the old evidence remains historically valid, requires qualification, or needs replacement through a new canonical workflow. Never “resolve” staleness by editing a label while leaving the evidence boundary ambiguous. When the object is a Comparison Direction, also check whether a **Potentially stale** label is only advisory. Dismiss the label if the team decides the direction still represents a useful boundary. {% example-demo title="Revised applicability after evaluation" %} Experts revise a Policy so it applies only when the customer explicitly requests a recommendation. Runs against the old Benchmark Version remain valid evidence under the former applicability rule. The current Dataset and linked Rubrics are reviewed, a new Snapshot and Benchmark Version establish the revised boundary, and new Runs use it. Any customer-owned human review context names both boundaries instead of marking every old result simply “wrong.” {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Versions, staleness, and resolution](/docs/object-model/versions-staleness-and-resolution) - [Staleness Detection](/docs/governance/staleness-detection) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Product quickstart](/docs/quickstart) - [Task index](/docs/operating-manual/task-index) {% /related-card-grid %} ## Source confidence Code-backed: current Policy and Rubric types preserve versioned governance facts, Snapshot routes preserve immutable benchmark evidence, and Comparison Directions expose a deliberately narrower advisory stale label. Cross-object staleness remains a review and routing decision, not an automatic global state transition.