# Assess trained candidates against an independent benchmark Generated: 2026-11-13T08:01:29.407Z Source build: local Canonical docs: https://teammately.ai/docs --- id: workflow.weights-training.independent-assessment title: Assess trained candidates against an independent benchmark summary: Measure actual trained-model behavior separately from generation checks and provider job status. kind: task product_area: weights_training status: stable updated: 2026-11-11 canonical: /docs/weights-training/independent-assessment --- # Assess trained candidates against an independent benchmark Measure actual trained-model behavior separately from generation checks and provider job status. ## Prerequisites An identifiable training output, a supported inference path, and an assessment population kept separate from training. ## Steps 1. Confirm the provider job completed and identify its resulting model. 2. Check how the selected inference connection or endpoint is associated with that output. 3. Authorize evaluation separately through the supported Results workflow. Inspect actual responses, rubric judgments, and completeness. 4. Compare against the intended baseline under interpretable benchmark and execution conditions. Examine both improvements and protected behavior. ## Object and state changes Training records a provider operation. Evaluation records the behavior of the identified trained candidate through its configured inference path. ## Success criteria A claimed improvement resolves to actual model behavior on the intended independent population. ## Common failure modes Using job completion as an evaluation; assessing an endpoint that still serves a different model; evaluating only on teaching examples. ## Related reference pages {% related-card-grid title="Continue this workflow" %} - [Export training packages](/docs/weights-training/exports-and-training) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Resolve a problem" %} - [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases) - [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage) {% /related-card-grid %} --- id: weights-training.exports-training title: Export training packages summary: Prepare an exact training package, launch an authorized provider job, and inspect a separately authorized held-out evaluation. kind: task product_area: weights_training status: stable updated: 2026-11-11 canonical: /docs/weights-training/exports-and-training --- # Export training packages Use Overview to prepare a dataset download and Results to configure connected training and evaluation. Downloading a package does not submit a training job. ## Prerequisites - A reviewed dataset with the intended splits. - An export profile compatible with its method. - An active provider connection for connected training. - An available supported model and explicit authorization to launch. ## Prepare the package Choose **Prepare download** in Overview. Inspect the pinned dataset, profile, counts and exclusions before creating the package. A ready package provides the canonical files and manifest for that exact dataset state. Canonical demonstration and preference exports preserve their method-specific targets. The package can include train, validation and test files where those splits have members, plus its manifest and supporting instructions/materials. A provider-specific conversion is a separate compatibility boundary. ## Train deliberately In Results choose the provider connection, base model and supported settings. Saving configuration alone does not upload data or start a job. Launch explicitly and follow the provider job identity and state. Choose an export and training profile that supports the selected method and material types. Confirm the connection’s model access, job permissions, and quota before launch. Compatibility checks identify any setting or dataset feature that needs attention. The training transfer uses train and validation as supported. Test content remains held out. Do not upload benchmark Cases or protected rubric/construction context as learner data. ## Evaluate separately Choose the inference connection or provider path for the trained model, then authorize evaluation. Record how the model is associated with that endpoint so the resulting assessment can be traced to the intended training output. Inspect the resulting model identity alongside actual responses, rubric outcomes, and evaluation completeness. ## Object and state changes Package creation captures the selected dataset. Launch records an authorized training operation and provider job. Cancellation and evaluation have separate operations. Existing packages and accepted results remain attributable. ## Success criteria - Package, dataset and provider job identities agree. - Held-out data stays out of training. - The resulting output is identified before evaluation. - Evaluation uses actual behavior rather than job status. ## Common failure modes - Assuming Save starts training. - Dropping unsupported fields to fit a provider. - Treating training completion as correctness. - Selecting an inference endpoint without checking its trained-model association. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Weights Training](/docs/weights-training) - [Training Datasets and Coverage](/docs/weights-training/datasets) - [Workspace Integrations](/docs/admin-console/integrations) - [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Missing Outputs](/docs/troubleshooting/missing-outputs) - [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs) {% /related-card-grid %} --- id: troubleshooting.unrealistic-synthetic-cases title: Synthetic Cases That Feel Unrealistic summary: Improve generated candidates whose surface details or behavior assumptions do not represent plausible target-system use. kind: error product_area: troubleshooting status: stable updated: 2026-11-06 canonical: /docs/troubleshooting/unrealistic-synthetic-cases --- # Synthetic Cases That Feel Unrealistic ## Symptoms - Generated language sounds artificial or describes a user who would not exist in the product. - Required context appears conveniently in the prompt even though the real system would retrieve it elsewhere. - The Case combines incompatible ontology values or violates product constraints. - Reviewers can identify the intended coverage tuple but would never admit the Case to a Benchmark. - Many generated candidates repeat the same template with superficial wording changes. ## Likely causes - The synthesis request lacks representative source Cases or product constraints. - A coverage tuple combines values that cannot coexist in the target system. - A Scenario or tuple is too broad, stale, or inconsistent with the target system. - Generated candidates are being treated as reviewed Benchmark Cases before admission. ## Diagnose the generation boundary 1. Open the intended Benchmark's **Coverage Design → Canvas** and identify the coverage slice used for Case preparation. Use **Assets → Cases** to import real examples or create a reusable Case manually. 2. Compare the generated Case with real in-use Cases from the same behavior region. 3. Separate a bad surface realization from a bad coverage target. Awkward wording may need editing; an impossible product state requires revised Scenario or tuple context. 4. Check whether the generator lacked constraints that operators hold only in private memory. 5. Inspect the synthesis lifecycle and candidate state. A completed generation is still candidate material, not a selected Benchmark Case. ## Fix - Add representative source Cases or explicit product constraints to the synthesis context. - Refine the Scenario or tuple when its construction guidance combines incompatible conditions. Keep response-alternative guidance in Comparison Directions. - Generate from a specific missing coverage tuple rather than asking for generic diversity. - Edit a promising candidate when its behavioral boundary is sound and only presentation is weak. - Reject candidates whose premise is impossible or whose source context cannot be justified. - Review duplicates before adding any generated Case to a Benchmark. Validate revised candidates with an expert who knows the target workflow. Ask whether the situation could occur, whether the system would receive the shown context, and whether the requested judgment matters. Only then classify and curate the Case for possible benchmark use. ## Prevention Ground synthesis in reviewed source Cases, explicit product constraints, and a named coverage gap. Review the Scenario and tuple, inspect duplicates, and require Case review before adding generated candidates to a Benchmark or Snapshot. ## Related task pages {% related-card-grid title="Related workflows" %} - [Synthesize Cases](/docs/coverage-engineering/synthesize-cases) - [Scenarios](/docs/coverage-engineering/scenarios) - [Boundary Cases](/docs/coverage-engineering/boundary-cases) - [Case Pool](/docs/coverage-engineering/case-pool) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference" %} - [Case Pool surface](/docs/object-model/case-pool) - [Cases](/docs/object-model/cases) - [Coverage dimensions](/docs/object-model/coverage-dimensions) {% /related-card-grid %} --- id: troubleshooting.unbalanced-coverage title: Unbalanced Coverage summary: Fix benchmarks that overrepresent easy or common cases while missing important behavior. kind: error product_area: troubleshooting status: stable updated: 2026-11-06 canonical: /docs/troubleshooting/unbalanced-coverage --- # Unbalanced Coverage Use this when the Benchmark Dataset overrepresents some behavior and misses other behavior that matters before human review. ## Symptom A benchmark looks strong in aggregate but reviewers or product owners identify important behavior areas with too few cases. Results may be dominated by common easy cases while rare or boundary cases are absent. ## Likely causes - Case import mirrored production volume instead of product risk. - Dimensions or metadata do not represent the behavior areas that matter. - Synthetic cases were added in one area but not reviewed across the full risk boundary. - Coverage was not refreshed after new signals, policy changes, or failures. ## Diagnostic checks - Inspect **Benchmark Datasets → Representation** and the Coverage Design overview by the configured Coverage Facets. - Look for policies or rubrics with very few applicable cases. - Compare production or expert signals with benchmark case distribution. - Inspect whether new cases are candidate-only or actually in use. ## Fix - Add representative, boundary, or adversarial cases for undercovered areas. - Refresh dimensions or metadata before judging balance. - Review and promote candidate cases into the benchmark only after context is complete. - Record remaining gaps as Coverage Stories or explicit benchmark coverage guidance. ## Prevention - Define coverage targets before benchmark runs. - Track candidate and in-use cases separately. - Review coverage after policy, rubric, or product behavior changes. - Do not use aggregate scores without a coverage note. ## Related task pages {% related-card-grid title="Related task pages" %} - [Dataset Representation](/docs/benchmark-datasets/representation) - [Coverage Design Settings](/docs/coverage-design/settings) - [Coverage Stories](/docs/coverage-design/coverage-stories) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Case Pool surface](/docs/object-model/case-pool) - [Coverage dimensions](/docs/object-model/coverage-dimensions) - [Benchmarks](/docs/object-model/benchmarks) {% /related-card-grid %}