# Training Datasets and Coverage Generated: 2026-11-03T08:07:08.108Z Source build: local Canonical docs: https://teammately.ai/docs --- id: weights-training.datasets title: Training Datasets and Coverage summary: Inspect generated training examples, preserve related families across splits, and assess representation from captured project facets. kind: task product_area: weights_training status: stable updated: 2026-10-30 canonical: /docs/weights-training/datasets --- # Training Datasets and Coverage Inspect the dataset before exporting or training. Generation completion does not establish realistic examples, balanced coverage or useful learned behavior. ## Prerequisites - A preparation tied to the intended Benchmark Version. - Generated examples or a supported existing dataset. - Clear training purpose and held-out evaluation intent. ## Inspect Dataset Open Dataset and review the input, response or preference target, captured criteria and observed judgments. Expected labels are not substitutes for the actual outcomes. Keep related construction-family members in the same split. A small input variation must not leak the same underlying situation across training and held-out testing. Inspect train, validation and test populations before packaging. Exclusion and split changes produce a new dataset state. Existing packages remain bound to their earlier captured content rather than silently acquiring those edits. ## Inspect Coverage Coverage uses the dataset's selected state and split filter. Read classified, no-match and unavailable outcomes distinctly. Unknown or unclassified examples remain in the denominator rather than disappearing from the population. New synthesis captures the project Dimension ontology, active Topics, Topic Groups and Scenarios used for classification. Retries reuse that vocabulary. A missing vocabulary is not a successful coverage assessment. Comparing with the captured benchmark can reveal a thin category, but the benchmark distribution is not silently imposed as the training distribution target. ## Object and state changes Dataset edits change current membership or split assignment through their owning commands. Representation analysis can publish a new dataset state without regenerating the input or response. It preserves recorded splits and captured vocabulary. ## Success criteria - Examples are suitable for the selected method and purpose. - Related families do not cross protected split boundaries. - Coverage counts include the unclassified population. - The exact dataset state used by a package is identifiable. ## Common failure modes - Treating generation status as a quality score. - Substituting expected judgments for observed results. - Reading missing classification as proven absence. - Training on the held-out test split. - Assuming a package changes after the dataset is edited. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Weights Training](/docs/weights-training) - [Export, Train, and Evaluate](/docs/weights-training/exports-and-training) - [Dataset Representation](/docs/benchmark-datasets/representation) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} --- id: assets.weights title: Weights Training summary: Prepare benchmark-guided training data, inspect coverage and splits, download exact packages, and follow training and evaluation results. kind: reference product_area: weights_training status: stable updated: 2026-10-30 canonical: /docs/weights-training --- # Weights Training Weights training turns selected Benchmark criteria into inspectable training material and connects that material to model training. Open it from the Benchmark workspace. ## Definition A preparation records the Benchmark Version, training method, selected scope and generation settings. The workspace has **Overview**, **Dataset**, **Coverage** and **Results**. Overview summarizes the preparation and prepares the exact dataset download. Dataset exposes examples and splits. Coverage shows representation. Results configures and follows connected training and separately authorized evaluation. ## Fields, states, or lifecycle rules - Demonstration and preference datasets have different learner targets. - Examples retain their governing criteria and source context. - Training, validation and test splits remain distinguishable. - A download package captures an exact dataset state and export profile. - A training run records the connection, base model, configuration, provider job and resulting model identity. - Training completion does not establish benchmark improvement. ## Use the workspace 1. Create a preparation for the intended Benchmark Version and method. 2. Inspect generated examples in Dataset. 3. Check splits and meaningful representation in Coverage. 4. Use **Prepare download** in Overview and inspect the package settings. 5. Download the exact ready package or use Results to configure a connected training service. 6. Launch training deliberately and follow the recorded job. 7. Authorize evaluation separately and inspect the actual resulting responses and judgments. A provider connection is not required to download a canonical dataset. Connected training services accept only the methods, model families and material types supported by their selected profile. ## Keep training and evaluation separate Generation checks describe prepared material. Coverage describes its representation. A provider completion identifies a training output. Held-out evaluation measures that output through its supported inference path. None substitutes for the others. Keep test records out of training and provider validation uploads. Preserve exact package and model identities when comparing results with an ordinary Harness evaluation. ## Related task pages {% related-card-grid title="Related task pages" %} - [Training Datasets and Coverage](/docs/weights-training/datasets) - [Export, Train, and Evaluate](/docs/weights-training/exports-and-training) - [Workspace Integrations](/docs/admin-console/integrations) - [Evaluation Results](/docs/benchmark-evaluations/results) {% /related-card-grid %} --- id: weights-training.exports-training title: Export, Train, and Evaluate summary: Prepare an exact training package, launch an authorized provider job, and inspect a separately authorized held-out evaluation. kind: task product_area: weights_training status: stable updated: 2026-10-30 canonical: /docs/weights-training/exports-and-training --- # Export, Train, and Evaluate Use Overview to prepare a dataset download and Results to configure connected training and evaluation. Downloading a package does not submit a training job. ## Prerequisites - A reviewed dataset with the intended splits. - An export profile compatible with its method. - An active provider connection for connected training. - An available supported model and explicit authorization to launch. ## Prepare the package Choose **Prepare download** in Overview. Inspect the pinned dataset, profile, counts and exclusions before creating the package. A ready package provides the canonical files and manifest for that exact dataset state. Canonical demonstration and preference exports preserve their method-specific targets. The package can include train, validation and test files where those splits have members, plus its manifest and supporting instructions/materials. A provider-specific conversion is a separate compatibility boundary. ## Train deliberately In Results choose the provider connection, base model and supported settings. Saving configuration alone does not upload data or start a job. Launch explicitly and follow the provider job identity and state. Provider profiles govern supported text, materials, tools, trajectories or preference formats. An unsupported feature is rejected rather than silently removed. A successful connection probe does not establish create permission, quota or successful training. The training transfer uses train and validation as supported. Test content remains held out. Do not upload benchmark Cases or protected rubric/construction context as learner data. ## Evaluate separately Authorize evaluation through the supported inference connection or provider path. Training does not automatically provision an inference deployment for every service. Where the deployment association is user-reported, keep that distinction visible. Inspect actual responses, rubric outcomes, completeness and the resulting model identity. A job that completed without a confirmed model output is not a verified trained candidate. ## Object and state changes Package creation captures the selected dataset. Launch records an authorized training operation and provider job. Cancellation and evaluation have separate operations. Existing packages and accepted results remain attributable. ## Success criteria - Package, dataset and provider job identities agree. - Held-out data stays out of training. - The resulting output is identified before evaluation. - Evaluation uses actual behavior rather than job status. ## Common failure modes - Assuming Save starts training. - Dropping unsupported fields to fit a provider. - Treating training completion as correctness. - Claiming inference resources were provisioned merely because training succeeded. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Weights Training](/docs/weights-training) - [Training Datasets and Coverage](/docs/weights-training/datasets) - [Workspace Integrations](/docs/admin-console/integrations) - [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Missing Outputs](/docs/troubleshooting/missing-outputs) - [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs) {% /related-card-grid %} --- id: benchmark-datasets.representation title: Dataset Representation summary: Analyze how distinct benchmark Cases are distributed across facets, evaluator rules, and provenance. kind: task product_area: benchmark_datasets status: stable updated: 2026-10-30 canonical: /docs/benchmark-datasets/representation --- # Dataset Representation ## Prerequisites - A current benchmark dataset or Snapshot with representation facts. - Coverage Facets and evaluator relationships meaningful enough to interpret. Representation groups the current or snapshotted dataset by governed facts. Available groupings include Dimension ontology values, Topic Groups, Topics, Scenarios, Policies, policy application, Rubrics, rubric application, presence of rubrics, and contributors. Choose **distinct Cases** when counts matter, or **Case share** when comparing proportions. Policy and rubric views can split by application state. Filters and drilldowns narrow the visible population, and the resulting table or chart can be exported as CSV. ## Reading the view - A large bar means concentration, not correctness. - An empty category can indicate a true coverage gap, an inactive facet, missing classification, or a filter that excludes the Cases. - Topic Groups do not merge their member Topics; group-level handling and Topic-level representation remain distinct. - Policy and rubric presence is not the same as approved eligible application. - Contributor distribution is provenance, not a substitute for agreement or evaluator quality. Use Coverage Design when a gap should drive a Coverage Story or Case Foundry work. Use Expert Contributions when the missing supporting information requires governed expert judgment. > Historical availability > > Representation is preserved when the Snapshot contains the required representation facts. Some older Snapshots may not expose this view; do not reconstruct their distribution from current mutable classifications. {% example-demo title="Example: count and share tell different stories" %} A Topic Group has twenty Cases but represents 60% of a small dataset, while a required ontology value has only two. Distinct count reveals the thin required value; Case share reveals the concentration. The operator records a Coverage Story instead of presenting the large Topic count as balanced coverage. {% /example-demo %} ## Object and state changes Grouping, metrics, filtering, splitting, drilldown, and CSV export change only the analysis view. They do not classify Cases, edit facets, or modify Snapshot content. ## Success criteria - Counts and shares use the intended Case population. - Missing, thin, and concentrated categories are distinguished. - A governed Coverage Story or follow-up owns any actionable gap. ## Common failure modes - Reading a filtered percentage as the whole dataset. - Equating high volume with representative coverage. - Reconstructing an old Snapshot from current classifications. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Dimensions and Ontology](/docs/coverage-engineering/dimensions-ontology) - [Topics](/docs/coverage-engineering/project-topics) - [Scenarios](/docs/coverage-engineering/scenarios) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) - [Stale Dimensions](/docs/troubleshooting/stale-dimensions) - [Dimension classification troubleshooting](/docs/troubleshooting/dimension-classification) {% /related-card-grid %} --- id: troubleshooting.unrealistic-synthetic-cases title: Synthetic Cases That Feel Unrealistic summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use. kind: error product_area: troubleshooting status: stable updated: 2026-10-30 canonical: /docs/troubleshooting/unrealistic-synthetic-cases --- # Synthetic Cases That Feel Unrealistic ## Symptoms - Generated language sounds artificial or describes a user who would not exist in the product. - Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere. - The Case combines incompatible ontology values or violates product constraints. - Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark. - Many generated candidates repeat the same template with superficial wording changes. ## Likely causes - The synthesis request lacks representative source Cases or product constraints. - A coverage tuple combines values that cannot coexist in the target system. - A Comparison Direction is too broad, stale, or duplicative. - Generation is being judged as finished benchmark results instead of candidate material. ## Diagnose the generation boundary 1. Identify the entry surface. **Assets → Cases → Synthesize** creates reusable project Case candidates; **Coverage Design → Case Foundry** coordinates candidates for a named benchmark need. 2. Compare the generated Case with real in-use Cases from the same behavior region. 3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires different inputs or directions. 4. Check whether the generator lacked constraints that operators hold only in private memory. 5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not selected Benchmark Dataset supporting information. ## Fix - Add representative source Cases or explicit product constraints to the synthesis context. - Narrow, edit, archive, or delete a Comparison Direction that produces implausible combinations. - Generate from a specific missing coverage tuple rather than asking for generic diversity. - Edit a promising candidate when its behavioral boundary is sound and only presentation is weak. - Reject candidates whose premise is impossible or whose source context cannot be justified. - Review duplicates before adding any generated Case to a Benchmark. Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use. ## Prevention Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Keep Comparison Directions narrow, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot. {% example-demo title="Example: impossible refund request" %} Synthesis produces a Case where an anonymous visitor requests an enterprise-contract refund and supplies internal approval metadata in chat. The tuple is valuable—exception request with conflicting authority—but the realization is impossible. The team adds authentication and channel constraints, narrows the Comparison Direction, and regenerates. The realistic candidate still goes through Case review before selection. {% /example-demo %} ## Related task pages {% related-card-grid title="Related workflows" %} - [Synthesize Cases](/docs/coverage-engineering/synthesize-cases) - [Comparison Directions](/docs/assets/comparison-directions) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Case Pool](/docs/coverage-engineering/case-pool) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Case Pool surface](/docs/object-model/case-pool) - [Cases](/docs/object-model/cases) - [Coverage dimensions](/docs/object-model/coverage-dimensions) {% /related-card-grid %} --- id: troubleshooting.unbalanced-coverage title: Unbalanced Coverage summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior. kind: error product_area: troubleshooting status: stable updated: 2026-10-30 canonical: /docs/troubleshooting/unbalanced-coverage --- # Unbalanced Coverage Use this when benchmark results overrepresents some behavior and misses other behavior that matters before human review. ## Symptom A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent. ## Likely causes - Case import mirrored production volume instead of product risk. - Dimensions or metadata do not represent the behavior areas that matter. - Synthetic cases were added in one area but not reviewed across the full risk boundary. - Coverage was not refreshed after new signals, policy changes, or failures. ## Diagnostic checks - Inspect **Benchmark Datasets → Representation** and the Coverage Design overview by the configured Coverage Facets. - Look for policies or rubrics with very few applicable cases. - Compare production or expert signals with benchmark case distribution. - Inspect whether new cases are candidate-only or actually in use. ## Fix - Add representative, boundary, or adversarial cases for undercovered areas. - Refresh dimensions or metadata before judging balance. - Review and promote candidate cases into the benchmark only after context is complete. - Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance. ## Prevention - Define coverage targets before benchmark runs. - Track candidate and in-use cases separately. - Review coverage after policy, rubric, or product behavior changes. - Do not use aggregate scores without a coverage note. ## Related task pages {% related-card-grid title="Related task pages" %} - [Dataset Representation](/docs/benchmark-datasets/representation) - [Coverage Design Settings](/docs/coverage-design/settings) - [Coverage Stories](/docs/coverage-design/coverage-stories) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Case Pool surface](/docs/object-model/case-pool) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %}