# Separate training, validation, and test data
Generated: 2026-11-13T08:01:24.765Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: workflow.weights-training.splits
title: Separate training, validation, and test data
summary: Keep related examples and protected assessment material from crossing the learner’s evaluation boundary.
kind: task
product_area: weights_training
status: stable
updated: 2026-11-11
canonical: /docs/weights-training/splits
---
# Separate training, validation, and test data
Keep related examples and protected assessment material from crossing the learner’s evaluation boundary.
## Prerequisites
A generated dataset with visible split assignments and construction-family relationships.
## Steps
1. Inspect the train, validation, and test populations in Dataset.
2. Keep members of the same construction family together so superficial variations cannot leak the same underlying task across splits.
3. Inspect exclusions and split changes in the resulting dataset state before packaging.
4. Confirm the training handoff uses only its permitted training and validation material. Preserve test content for separate assessment.
## Object and state changes
Split and exclusion edits produce a new dataset state. Existing packages retain the state they captured.
## Success criteria
The exact package identifies the reviewed split populations and held-out material stays outside provider training uploads.
## Common failure modes
Randomly separating close variants without inspecting their shared construction; assuming a package follows later dataset edits.
## Related reference pages
{% related-card-grid title="Continue this workflow" %}
- [Inspect dataset quality and coverage](/docs/weights-training/datasets)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Resolve a problem" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage)
{% /related-card-grid %}
---
id: weights-training.datasets
title: Inspect dataset quality and coverage
summary: Inspect generated training examples, preserve related families across splits, and assess representation from captured project facets.
kind: task
product_area: weights_training
status: stable
updated: 2026-11-11
canonical: /docs/weights-training/datasets
---
# Inspect dataset quality and coverage
Inspect the dataset before exporting or training. Generation completion does not establish realistic examples, balanced coverage or useful learned behavior.
## Prerequisites
- A preparation tied to the intended Benchmark Version.
- Generated examples or a supported existing dataset.
- Clear training purpose and held-out evaluation intent.
## Inspect Dataset
Open Dataset and review the input, response or preference target, captured criteria and observed judgments. Expected labels are not substitutes for the actual outcomes.
Keep related construction-family members in the same split. A small input variation must not leak the same underlying situation across training and held-out testing. Inspect train, validation and test populations before packaging.
Exclusion and split changes produce a new dataset state. Existing packages remain bound to their earlier captured content rather than silently acquiring those edits.
## Inspect Coverage
Coverage uses the dataset's selected state and split filter. Read classified, no-match and unavailable outcomes distinctly. Unknown or unclassified examples remain in the denominator rather than disappearing from the population.
New synthesis captures the project Dimension ontology, active Topics, Topic Groups and Scenarios used for classification. Retries reuse that vocabulary. A missing vocabulary is not a successful coverage assessment.
Comparing with the captured benchmark can reveal a thin category, but the benchmark distribution is not silently imposed as the training distribution target.
## Object and state changes
Dataset edits change current membership or split assignment through their owning commands. Representation analysis can publish a new dataset state without regenerating the input or response. It preserves recorded splits and captured vocabulary.
## Success criteria
- Examples are suitable for the selected method and purpose.
- Related families do not cross protected split boundaries.
- Coverage counts include the unclassified population.
- The exact dataset state used by a package is identifiable.
## Common failure modes
- Treating generation status as a quality score.
- Substituting expected judgments for observed results.
- Reading missing classification as proven absence.
- Training on the held-out test split.
- Assuming a package changes after the dataset is edited.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Weights Training](/docs/weights-training)
- [Export, Train, and Evaluate](/docs/weights-training/exports-and-training)
- [Dataset Representation](/docs/benchmark-datasets/representation)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage)
{% /related-card-grid %}
---
id: troubleshooting.unrealistic-synthetic-cases
title: Synthetic Cases That Feel Unrealistic
summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/unrealistic-synthetic-cases
---
# Synthetic Cases That Feel Unrealistic
## Symptoms
- Generated language sounds artificial or describes a user who would not exist in the product.
- Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere.
- The Case combines incompatible ontology values or violates product constraints.
- Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark.
- Many generated candidates repeat the same template with superficial wording changes.
## Likely causes
- The synthesis request lacks representative source Cases or product constraints.
- A coverage tuple combines values that cannot coexist in the target system.
- A Scenario or tuple is too broad, stale, or inconsistent with the target system.
- Generated candidates are being treated as reviewed Benchmark Cases before admission.
## Diagnose the generation boundary
1. Open the intended Benchmark's **Coverage Design → Canvas** and identify the coverage slice used for Case preparation. Use **Assets → Cases** to import real examples or create a reusable Case manually.
2. Compare the generated Case with real in-use Cases from the same behavior region.
3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires revised Scenario or tuple context.
4. Check whether the generator lacked constraints that operators hold only in private memory.
5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not a selected Benchmark Case.
## Fix
- Add representative source Cases or explicit product constraints to the synthesis context.
- Refine the Scenario or tuple when its construction guidance combines incompatible conditions. Keep response-alternative guidance in Comparison Directions.
- Generate from a specific missing coverage tuple rather than asking for generic diversity.
- Edit a promising candidate when its behavioral boundary is sound and only presentation is weak.
- Reject candidates whose premise is impossible or whose source context cannot be justified.
- Review duplicates before adding any generated Case to a Benchmark.
Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use.
## Prevention
Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Review the Scenario and tuple, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot.
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Scenarios](/docs/coverage-engineering/scenarios)
- [Boundary Cases](/docs/coverage-engineering/boundary-cases)
- [Case Pool](/docs/coverage-engineering/case-pool)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Case Pool surface](/docs/object-model/case-pool)
- [Cases](/docs/object-model/cases)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
{% /related-card-grid %}
---
id: troubleshooting.unbalanced-coverage
title: Unbalanced Coverage
summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/unbalanced-coverage
---
# Unbalanced Coverage
Use this when the Benchmark Dataset overrepresents some behavior and misses other behavior that matters before human review.
## Symptom
A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent.
## Likely causes
- Case import mirrored production volume instead of product risk.
- Dimensions or metadata do not represent the behavior areas that matter.
- Synthetic cases were added in one area but not reviewed across the full risk boundary.
- Coverage was not refreshed after new signals, policy changes, or failures.
## Diagnostic checks
- Inspect **Benchmark Datasets → Representation** and the Coverage Design overview by the configured Coverage Facets.
- Look for policies or rubrics with very few applicable cases.
- Compare production or expert signals with benchmark case distribution.
- Inspect whether new cases are candidate-only or actually in use.
## Fix
- Add representative, boundary, or adversarial cases for undercovered areas.
- Refresh dimensions or metadata before judging balance.
- Review and promote candidate cases into the benchmark only after context is complete.
- Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance.
## Prevention
- Define coverage targets before benchmark runs.
- Track candidate and in-use cases separately.
- Review coverage after policy, rubric, or product behavior changes.
- Do not use aggregate scores without a coverage note.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Dataset Representation](/docs/benchmark-datasets/representation)
- [Coverage Design Settings](/docs/coverage-design/settings)
- [Coverage Stories](/docs/coverage-design/coverage-stories)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Case Pool surface](/docs/object-model/case-pool)
- [Coverage dimensions](/docs/object-model/coverage-dimensions)
- [Benchmarks](/docs/object-model/benchmarks)
{% /related-card-grid %}