Training Datasets and Coverage
Inspect the dataset before exporting or training. Generation completion does not establish realistic examples, balanced coverage or useful learned behavior.
Prerequisites
- A preparation tied to the intended Benchmark Version.
- Generated examples or a supported existing dataset.
- Clear training purpose and held-out evaluation intent.
Inspect Dataset
Open Dataset and review the input, response or preference target, captured criteria and observed judgments. Expected labels are not substitutes for the actual outcomes.
Keep related construction-family members in the same split. A small input variation must not leak the same underlying situation across training and held-out testing. Inspect train, validation and test populations before packaging.
Exclusion and split changes produce a new dataset state. Existing packages remain bound to their earlier captured content rather than silently acquiring those edits.
Inspect Coverage
Coverage uses the dataset's selected state and split filter. Read classified, no-match and unavailable outcomes distinctly. Unknown or unclassified examples remain in the denominator rather than disappearing from the population.
New synthesis captures the project Dimension ontology, active Topics, Topic Groups and Scenarios used for classification. Retries reuse that vocabulary. A missing vocabulary is not a successful coverage assessment.
Comparing with the captured benchmark can reveal a thin category, but the benchmark distribution is not silently imposed as the training distribution target.
Object and state changes
Dataset edits change current membership or split assignment through their owning commands. Representation analysis can publish a new dataset state without regenerating the input or response. It preserves recorded splits and captured vocabulary.
Success criteria
- Examples are suitable for the selected method and purpose.
- Related families do not cross protected split boundaries.
- Coverage counts include the unclassified population.
- The exact dataset state used by a package is identifiable.
Common failure modes
- Treating generation status as a quality score.
- Substituting expected judgments for observed results.
- Reading missing classification as proven absence.
- Training on the held-out test split.
- Assuming a package changes after the dataset is edited.