# Export, Train, and Evaluate

Generated: 2026-11-02T05:44:01.419Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="weights-training.exports-training" slug="weights-training/exports-and-training" url="https://teammately.ai/docs/weights-training/exports-and-training.md" title="Export, Train, and Evaluate" updated="2026-10-30" status="stable">
---
id: weights-training.exports-training
title: Export, Train, and Evaluate
summary: Prepare an exact training package, launch an authorized provider job, and inspect a separately authorized held-out evaluation.
kind: task
product_area: weights_training
status: stable
updated: 2026-10-30
canonical: /docs/weights-training/exports-and-training
---

# Export, Train, and Evaluate

Use Overview to prepare a dataset download and Results to configure connected training and evaluation. Downloading a package does not submit a training job.

## Prerequisites

- A reviewed dataset with the intended splits.
- An export profile compatible with its method.
- An active provider connection for connected training.
- An available supported model and explicit authorization to launch.

## Prepare the package

Choose **Prepare download** in Overview. Inspect the pinned dataset, profile, counts and exclusions before creating the package. A ready package provides the canonical files and manifest for that exact dataset state.

Canonical demonstration and preference exports preserve their method-specific targets. The package can include train, validation and test files where those splits have members, plus its manifest and supporting instructions/materials. A provider-specific conversion is a separate compatibility boundary.

## Train deliberately

In Results choose the provider connection, base model and supported settings. Saving configuration alone does not upload data or start a job. Launch explicitly and follow the provider job identity and state.

Provider profiles govern supported text, materials, tools, trajectories or preference formats. An unsupported feature is rejected rather than silently removed. A successful connection probe does not establish create permission, quota or successful training.

The training transfer uses train and validation as supported. Test content remains held out. Do not upload benchmark Cases or protected rubric/construction context as learner data.

## Evaluate separately

Authorize evaluation through the supported inference connection or provider path. Training does not automatically provision an inference deployment for every service. Where the deployment association is user-reported, keep that distinction visible.

Inspect actual responses, rubric outcomes, completeness and the resulting model identity. A job that completed without a confirmed model output is not a verified trained candidate.

## Object and state changes

Package creation captures the selected dataset. Launch records an authorized training operation and provider job. Cancellation and evaluation have separate operations. Existing packages and accepted results remain attributable.

## Success criteria

- Package, dataset and provider job identities agree.
- Held-out data stays out of training.
- The resulting output is identified before evaluation.
- Evaluation uses actual behavior rather than job status.

## Common failure modes

- Assuming Save starts training.
- Dropping unsupported fields to fit a provider.
- Treating training completion as correctness.
- Claiming inference resources were provisioned merely because training succeeded.
## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Weights Training](/docs/weights-training)
- [Training Datasets and Coverage](/docs/weights-training/datasets)
- [Workspace Integrations](/docs/admin-console/integrations)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Missing Outputs](/docs/troubleshooting/missing-outputs)
- [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs)
{% /related-card-grid %}
</doc>

<doc id="assets.weights" slug="weights-training" url="https://teammately.ai/docs/weights-training.md" title="Weights Training" updated="2026-10-30" status="stable">
---
id: assets.weights
title: Weights Training
summary: Prepare benchmark-guided training data, inspect coverage and splits, download exact packages, and follow training and evaluation results.
kind: reference
product_area: weights_training
status: stable
updated: 2026-10-30
canonical: /docs/weights-training
---

# Weights Training

Weights training turns selected Benchmark criteria into inspectable training material and connects that material to model training. Open it from the Benchmark workspace.

## Definition

A preparation records the Benchmark Version, training method, selected scope and generation settings. The workspace has **Overview**, **Dataset**, **Coverage** and **Results**.

Overview summarizes the preparation and prepares the exact dataset download. Dataset exposes examples and splits. Coverage shows representation. Results configures and follows connected training and separately authorized evaluation.

## Fields, states, or lifecycle rules

- Demonstration and preference datasets have different learner targets.
- Examples retain their governing criteria and source context.
- Training, validation and test splits remain distinguishable.
- A download package captures an exact dataset state and export profile.
- A training run records the connection, base model, configuration, provider job and resulting model identity.
- Training completion does not establish benchmark improvement.

## Use the workspace

1. Create a preparation for the intended Benchmark Version and method.
2. Inspect generated examples in Dataset.
3. Check splits and meaningful representation in Coverage.
4. Use **Prepare download** in Overview and inspect the package settings.
5. Download the exact ready package or use Results to configure a connected training service.
6. Launch training deliberately and follow the recorded job.
7. Authorize evaluation separately and inspect the actual resulting responses and judgments.

A provider connection is not required to download a canonical dataset. Connected training services accept only the methods, model families and material types supported by their selected profile.

## Keep training and evaluation separate

Generation checks describe prepared material. Coverage describes its representation. A provider completion identifies a training output. Held-out evaluation measures that output through its supported inference path. None substitutes for the others.

Keep test records out of training and provider validation uploads. Preserve exact package and model identities when comparing results with an ordinary Harness evaluation.
## Related task pages

{% related-card-grid title="Related task pages" %}
- [Training Datasets and Coverage](/docs/weights-training/datasets)
- [Export, Train, and Evaluate](/docs/weights-training/exports-and-training)
- [Workspace Integrations](/docs/admin-console/integrations)
- [Evaluation Results](/docs/benchmark-evaluations/results)
{% /related-card-grid %}
</doc>

<doc id="weights-training.datasets" slug="weights-training/datasets" url="https://teammately.ai/docs/weights-training/datasets.md" title="Training Datasets and Coverage" updated="2026-10-30" status="stable">
---
id: weights-training.datasets
title: Training Datasets and Coverage
summary: Inspect generated training examples, preserve related families across splits, and assess representation from captured project facets.
kind: task
product_area: weights_training
status: stable
updated: 2026-10-30
canonical: /docs/weights-training/datasets
---

# Training Datasets and Coverage

Inspect the dataset before exporting or training. Generation completion does not establish realistic examples, balanced coverage or useful learned behavior.

## Prerequisites

- A preparation tied to the intended Benchmark Version.
- Generated examples or a supported existing dataset.
- Clear training purpose and held-out evaluation intent.

## Inspect Dataset

Open Dataset and review the input, response or preference target, captured criteria and observed judgments. Expected labels are not substitutes for the actual outcomes.

Keep related construction-family members in the same split. A small input variation must not leak the same underlying situation across training and held-out testing. Inspect train, validation and test populations before packaging.

Exclusion and split changes produce a new dataset state. Existing packages remain bound to their earlier captured content rather than silently acquiring those edits.

## Inspect Coverage

Coverage uses the dataset's selected state and split filter. Read classified, no-match and unavailable outcomes distinctly. Unknown or unclassified examples remain in the denominator rather than disappearing from the population.

New synthesis captures the project Dimension ontology, active Topics, Topic Groups and Scenarios used for classification. Retries reuse that vocabulary. A missing vocabulary is not a successful coverage assessment.

Comparing with the captured benchmark can reveal a thin category, but the benchmark distribution is not silently imposed as the training distribution target.

## Object and state changes

Dataset edits change current membership or split assignment through their owning commands. Representation analysis can publish a new dataset state without regenerating the input or response. It preserves recorded splits and captured vocabulary.

## Success criteria

- Examples are suitable for the selected method and purpose.
- Related families do not cross protected split boundaries.
- Coverage counts include the unclassified population.
- The exact dataset state used by a package is identifiable.

## Common failure modes

- Treating generation status as a quality score.
- Substituting expected judgments for observed results.
- Reading missing classification as proven absence.
- Training on the held-out test split.
- Assuming a package changes after the dataset is edited.
## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Weights Training](/docs/weights-training)
- [Export, Train, and Evaluate](/docs/weights-training/exports-and-training)
- [Dataset Representation](/docs/benchmark-datasets/representation)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unbalanced Coverage](/docs/troubleshooting/unbalanced-coverage)
{% /related-card-grid %}
</doc>

<doc id="admin-console.integrations" slug="admin-console/integrations" url="https://teammately.ai/docs/admin-console/integrations.md" title="Workspace Integrations" updated="2026-10-30" status="stable">
---
id: admin-console.integrations
title: Workspace Integrations
summary: Connect messaging, repository, trace-store, and training services through their own authorization and operating controls.
kind: reference
product_area: admin_console
status: stable
updated: 2026-10-30
canonical: /docs/admin-console/integrations
---

# Workspace Integrations

## Definition

Open **Admin → Integrations** to connect workspace services. Each connection supplies a specific workflow rather than a general permission to read or write every external resource.

## Fields, states, or lifecycle rules

- Messaging connections support their configured notification and test operations.
- Repository connections discover repositories accessible to the authorizing account.
- Trace-store connections supply their supported capture/import workflow.
- Training connections are selected separately by a Benchmark preparation.
- Connection state, account authorization and a successful downstream operation remain distinct.

## Messaging

Slack and Microsoft Teams have provider-specific connection controls. For Teams, installation and a personal-message test are distinct operations. Workspace settings alone do not confirm installation or that automatic notifications are enabled.

Read provider errors as operational outcomes rather than treating a visible integration card as proof of delivery.

## Repositories and trace stores

GitHub.com, GitLab.com and Bitbucket Cloud connections discover the repositories allowed by the account and installation. Bind the intended repository through Reference Materials and inspect its indexed content.

LangSmith and Langfuse have their own connection surfaces. Keep original record identity and scope when importing material; connected access does not approve the captured behavior.

## Training and inference

Weights training selects an active service connection, supported model and compatible method. Connection checks do not certify quota, create-job permission, storage access or training quality.

Some evaluation paths require an existing callable inference deployment. A training job does not automatically allocate serving resources for every service. The deployment-to-output association can be user-reported where the adapter identifies it that way.

## Credentials and resource boundaries

For setup steps and provider-specific recovery, use the [integration guides](/docs/integrations#connection-setup-guides). Each guide distinguishes workspace connection, Project selection and a successful downstream operation.

Use the provider's owning connection flow and approved secret mechanism. Keep access scoped to the authorized account, identity and resource. Existing Vertex and Baseten training adapters do not authorize Google platform storage, ambient developer credentials or a provider fallback.

A failed service operation should remain actionable in its owning workflow rather than being hidden by a substitute local service.
## Related task pages

{% related-card-grid title="Related task pages" %}
- [Reference Materials](/docs/agent-setup/reference-materials)
- [Export, Train, and Evaluate](/docs/weights-training/exports-and-training)
- [Fieldloop](/docs/fieldloop)
- [Admin Console](/docs/admin-console)
{% /related-card-grid %}
</doc>

<doc id="benchmark-evaluations.inspect" slug="benchmark-evaluations/inspect-results" url="https://teammately.ai/docs/benchmark-evaluations/inspect-results.md" title="Inspect Evaluation Results" updated="2026-10-30" status="stable">
---
id: benchmark-evaluations.inspect
title: Inspect Evaluation Results
summary: Trace scores to exact responses and rubric judgments, distinguish incomplete results, and record authorized corrections.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-10-30
canonical: /docs/benchmark-evaluations/inspect-results
---

# Inspect Evaluation Results

Start with identity and completeness, then explain the score from the actual response and applicable Rubrics.

## Prerequisites

- Access to the selected Benchmark Version and Run.
- Available responses or evaluation progress.
- Project-management permission if a judgment correction is needed.

## Steps

1. Open Runs and confirm the candidate, saved Harness Version where present, machine and planned Trials.
2. Identify whether this is the Run used for score or another independent execution.
3. Open Results and inspect the Case population, applicable criteria and incomplete count.
4. Read failed gates, earned positive points and triggered penalties separately.
5. Open the response and rubric detail. Compare the criterion with the actual behavior and visible Case materials.
6. Inspect individual Trial outcomes when reading a repeated result.
7. Decide whether the responsible change belongs to candidate code, Case content, applicability, rubric wording or the observed judgment.

## Scores and missing results

Point scores can be negative, including below -100% for mixed positive/penalty Cases. A completed mandatory-gate failure contributes zero to the benchmark score. Missing or incompatible judgments stay incomplete even when another gate has failed.

The full planned population remains the denominator. An incomplete evaluation cannot be made comparable by ignoring its difficult or failed executions.

## Correct a judgment

An authorized project manager can choose **Correct judgment** on the exact Trial's rubric result and supply the supported corrected judgment and rationale. The original AI judgment and manual change remain recorded. Reopen the refreshed result to interpret its resulting scores consistently.

Changing the judgment is different from editing the Rubric or the Case. Correct the owning artifact when its meaning is wrong.

## Object and state changes

Inspection and filtering do not mutate results. A judgment correction records an attributable authoritative change. Starting a Contribution or an improvement Session creates separate work and preserves the inspected Run.

## Success criteria

- The conclusion identifies its Run, Trial and Benchmark Version.
- Scores are explained by applicable criteria and the configured Regime.
- Missing telemetry and judgments remain unknown.
- Corrections retain original judgment provenance.

## Common failure modes

- Reporting a rank without completeness.
- Assuming all results are read-only.
- Treating a gate failure as permission to replace unavailable results with zero.
- Inferring hidden model reasoning from a captured trajectory.
## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
- [Rubric Points and Benchmark Scores](/docs/correctness-governance/scoring)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Overlapping Rubrics](/docs/troubleshooting/overlapping-rubrics)
{% /related-card-grid %}
</doc>

<doc id="troubleshooting.missing-outputs" slug="troubleshooting/missing-outputs" url="https://teammately.ai/docs/troubleshooting/missing-outputs.md" title="Missing Outputs" updated="2026-10-30" status="stable">
---
id: troubleshooting.missing-outputs
title: Missing Outputs
summary: Separate a missing managed Run response from an unmatched row in an imported output-only Run.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/missing-outputs
---

# Missing Outputs

Use this when an Evaluation Run has no response for an intended Case. First identify whether the supporting information came from managed Harness execution or an imported output-only Run.

## Symptom

A managed Run is incomplete or has a failed or unevaluable Case attempt, or an imported output-only Run shows an unmatched Case row. An input-only Case by itself is not an error: managed execution is expected to generate its response.

## Likely causes

- Managed execution failed, stopped, or could not evaluate the Case.
- The wrong saved Harness Version or execution settings were selected.
- An external output row has a missing, unknown, or duplicate immutable Case ID.
- The imported file omitted Cases that the output-only Run was intended to cover.

## Diagnostic checks

- Open the Run and check its lifecycle, attempts, failed Cases, and selected Harness Version.
- Confirm that the Case belongs to the immutable Benchmark Version used by the Run.
- For imported outputs, compare external identifiers with the immutable Case IDs shown by the mapping surface.
- Confirm that the intended output column and candidate metadata were selected before import.

## Fix

- For managed execution, correct the Harness or execution problem and start a new Run against the same Benchmark Version when that boundary is still intended.
- For imported outputs, correct the external rows and repeat mapping with immutable Case IDs.
- Keep managed and imported-output diagnoses separate; an imported response does not become a saved Harness Version.
- Change the current Dataset and create a new Snapshot only when Case membership itself is wrong—not merely because a Run response is missing.

## Prevention

- Save the exact Harness Version before managed execution and review execution settings.
- Export or preserve immutable Case IDs through external generation.
- Validate several mapped rows before creating an output-only Run.
- Use Run Metadata to identify the candidate or external output set without pretending it is a Harness Version.

## Related task pages

{% related-card-grid title="Related task pages" %}
- [Import and prepare cases](/docs/operating-manual/import-and-prepare-cases)
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Outputs](/docs/object-model/outputs)
- [Cases](/docs/assets/cases)
- [Benchmarks](/docs/object-model/benchmarks)
{% /related-card-grid %}
</doc>

<doc id="troubleshooting.benchmark-runs" slug="troubleshooting/benchmark-runs" url="https://teammately.ai/docs/troubleshooting/benchmark-runs.md" title="Benchmark run troubleshooting" updated="2026-10-30" status="stable">
---
id: troubleshooting.benchmark-runs
title: Benchmark run troubleshooting
summary: Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/benchmark-runs
---

# Benchmark run troubleshooting

## Symptoms

- The evaluation surface has no Benchmark Version to run.
- Output import or mapping cannot identify a Case or output column.
- A Run is created but remains empty, incomplete, or failed.
- Results appear under the wrong candidate label or Harness Version.
- Two Runs show a score change but do not share a comparable snapshot boundary.

## Likely causes

- No immutable Snapshot exists for the intended evaluation set.
- Imported outputs are mapped to the wrong Case or column.
- The saved Harness Version or candidate metadata does not match the evaluated system.
- The Run is partial, failed, or being compared across different Benchmark Versions.

## Check the boundary before the failure

1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version.
2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run.
3. In Run setup, confirm the saved Harness Version and candidate metadata describe the system that produced the outputs.
4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior.
5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates.

## Fix

- **No runnable version:** finish Case selection and create a Snapshot first.
- **No mapped outputs:** correct Case identifiers or column mapping, then submit again under the intended Run.
- **Wrong Harness or metadata:** create a correctly configured Run. Do not relabel completed supporting information to represent a different system.
- **Partial failure:** preserve successful per-Case supporting information when the product does, correct the failed input or execution boundary, and rerun using a clearly named attempt.
- **Confusing comparison:** compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed.

## Prevention

Create a Snapshot before execution, save the exact Harness Version, preview output mapping on representative Cases, and name candidate metadata consistently. Before comparing, confirm that every difference between the two Runs is intentional and visible.

## Recovery check

Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping.

{% example-demo title="Example: scores drop after output import" %}
A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version.
{% /example-demo %}

## Related task pages

{% related-card-grid title="Related workflows" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Outputs](/docs/object-model/outputs)
{% /related-card-grid %}
</doc>