Stale Dimensions
Symptoms
- New Cases repeatedly fall into other, unknown, or no value.
- An ontology value refers to a product state or source hierarchy that no longer exists.
- Important failures concentrate in metadata that no Dimension represents.
- A coverage plan looks balanced under old labels but reviewers describe a new boundary.
- Two values have become indistinguishable after a product change.
Likely causes
- Product behavior or source authority changed while the coverage vocabulary did not.
- New Cases reveal an axis the existing schema never represented.
- Ontology values were renamed or repurposed without reviewing old classifications.
- The apparent staleness is actually incomplete Case classification.
Confirm staleness
First distinguish a stale schema from incomplete classification. Sample new and old Cases using the current Dimension definition. If the existing values still describe the behavior and only new Cases are unlabeled, repair classification. If reviewers need a new concept, different source authority, or changed applicability to classify consistently, the Dimension or ontology may be stale.
Fix
- Record the change that made the current vocabulary inadequate.
- Inspect the Dimension definition, values, examples, origin, and where it is used in coverage plans.
- Decide whether to rename a value, add a value, split the Dimension, replace it, or preserve it with a historical time boundary.
- Review representative Cases against the proposed schema before broad reclassification.
- Reclassify affected current Cases and inspect whether coverage gaps or target distributions changed.
- Update Coverage Management → Get Started and the overview deliberately. Do not change selected Dataset membership merely to preserve an old-looking distribution.
- Create a new Snapshot when the classification or selected evidence boundary used by the Benchmark changes.
Historical Snapshots and Runs should retain their original interpretation. A new Dimension schema can supersede the current planning model without making the old model disappear.
Prevention
Review Dimension definitions alongside product and source changes, keep representative examples for each ontology value, and inspect unclassified or catch-all Cases regularly. Name the schema and Snapshot boundary used when segment results inform a decision.
Worked example
Example: channel labels stop explaining escalation risk
A support Benchmark classifies Cases only by email and chat. After voice transcripts arrive, experts find that synchronous versus asynchronous interaction—not channel name—explains escalation behavior. The team creates a clearer interaction-mode Dimension, samples old and new Cases, updates Coverage Management, and creates a new Snapshot. Older results remain labeled under the prior schema.
Source confidence
Code-backed: Dimension types, Dimensions and Ontology, Coverage Management, and Get Started show the editable vocabulary and its use in benchmark planning. The product does not automatically prove conceptual staleness; the trigger comes from changed evidence and reviewer interpretation.