# Start an Improvement Session

Generated: 2026-09-13T04:39:07.534Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="improve.start-session" slug="improve/start-improvement-session" url="https://teammately.ai/docs/improve/start-improvement-session.md" title="Start an Improvement Session" updated="2026-09-07" status="stable">
---
id: improve.start-session
title: Start an Improvement Session
summary: Start from benchmark evidence, prepare a measurable Goal Contract, and choose bounded Work or Evolve behavior.
kind: task
product_area: improve
status: stable
updated: 2026-09-07
canonical: /docs/improve/start-improvement-session
---

# Start an Improvement Session

Start an Improvement Session when evaluation evidence justifies a candidate change or bounded investigation. The setup should turn a free-form intention into a measurable Goal Contract before work begins.

Choose the saved Harness Version and a specific baseline launch. A baseline may contain one Run. Set the number of Runs for future candidate evaluations independently; a difference from the baseline count is informational. The work forecast and authorized budget use the chosen candidate count.

## Prerequisites

- A selected benchmark and evidence that identifies the relevant Benchmark Version or Run.
- An existing target Harness and saved starting version.
- Case, rubric, Run, comparison, or frontier evidence that explains the need.
- A measurable outcome and constraints that should remain protected.
- An operator authorized to start and control the session.

## Steps

1. Open the selected benchmark and choose **Improve**.
2. Choose the available Improve experience, then create a new Improvement Session and select **Work** or **Evolve** when using Coevolve.
3. Select the target Harness, exact starting Harness Version, starting Run, and execution source.
4. For Work, choose the available Coevolve or External Agents path. For Evolve, use Coevolve and choose user-gated or autonomous execution. External fine-tuning has its own provider and return boundary when available.
5. State the desired behavior change and important non-regression constraints.
6. Prepare the Goal Contract. Resolve canonical target identities, objectives, measurement bindings, intervention constraints, and unresolved items.
7. Inspect the proposed revision and confirm it only when the evidence can measure the requested outcome.
8. For Evolve, configure epoch authorization, Case pass target, and provider usage bounds before starting.
9. If using an external worker, verify the scoped package and return contract after the Goal is confirmed.
10. Start the session and use chronology, trajectories, candidates, receipts, and current frontier to follow observable progress.

## Object and state changes

This task creates a benchmark-scoped Improvement Session, records its mode, experience, source, target, and pinned starting evidence, and establishes a Goal Contract revision. Session lifecycle states are draft, active, paused, completing, completed, cancelling, cancelled, or failed, with attention states when operator action is needed. Starting work can create worker packages, candidate Harness Versions, canonical evaluation requests and receipts, frontier changes, chronology events, and usage records.

## Success criteria

- The target and starting evidence use canonical identities.
- Every objective has an observable measurement binding.
- Constraints protect important behavior from hidden regression.
- Work or Evolve is chosen deliberately.
- The exact starting Harness Version and Run are visible.
- Candidate progress is supported by returned artifacts and evaluation receipts.
- The current frontier is explainable from the Goal Contract and evidence.

## Common failure modes

- Starting from an aggregate score without selected case or rubric evidence.
- Confirming a Goal Contract whose outcome cannot be measured.
- Allowing Evolve without bounded authorization.
- Treating a worker package as proof that private work occurred.
- Retaining the newest candidate without checking constraints and regressions.
- Changing the benchmark boundary during the session without making the new evidence explicit.

{% example-demo title="Example: bounded Work session" %}
A Run fails three cases because the Harness uses a superseded source. The operator pins those cases and the grounding rubric, targets the exact saved Harness version, and writes a Goal Contract requiring current-source selection without reducing missing-source uncertainty performance. Work begins only after both objectives have measurement bindings.
{% /example-demo %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Improve](/docs/improve)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Unbalanced coverage](/docs/troubleshooting/unbalanced-coverage)
{% /related-card-grid %}

## Source confidence

Code-backed: the current Improve setup, session command, and Goal Contract behavior support this workflow. External workers remain bounded by observable return artifacts and requests.
</doc>

<doc id="improve.overview" slug="improve" url="https://teammately.ai/docs/improve.md" title="Improve" updated="2026-09-07" status="stable">
---
id: improve.overview
title: Improve
summary: Coordinate durable Improvement Sessions from pinned benchmark evidence through Goal Contracts, candidates, evaluations, trajectories, and frontiers.
kind: concept
product_area: improve
status: stable
updated: 2026-09-07
canonical: /docs/improve
---

# Improve

Improve is the benchmark-scoped workspace for coordinating candidate work from benchmark evidence. Each Improvement Session pins the exact starting evidence it uses, including any Benchmark Version, Harness Version, and Run identities. Sessions preserve the goal, authorization, agent and worker activity, candidate Harness Versions, canonical evaluation receipts, and Current frontier.

## Definition

A new session begins with **Start from evidence**. The operator selects the target and relevant benchmark evidence, then prepares a **Goal Contract**. The contract binds the free-form objective to canonical target identities, measurement bindings, constraints, unresolved items, and intervention boundaries.

The current Improve workspace offers **Coevolve**, **External Agents**, and **External fine-tuning** experiences where enabled. Coevolve supports bounded Work and Evolve behavior; external experiences use explicit handoff or provider boundaries. All retained candidates still rely on observable Harness Versions and canonical Benchmark Evaluations rather than unrecorded claims of progress.

## Decision checkpoint

| Need | Session choice | Required evidence |
| --- | --- | --- |
| Investigate or implement one bounded direction | Work | Confirmed Goal Contract and pinned failures or target evidence |
| Explore several competing hypotheses | Evolve | Comparable measurement plan and explicit exploration authorization |
| Delegate implementation externally | Worker package | Pinned target, return contract, and observable returned candidate |
| Decide which candidates remain viable | Current frontier | Canonical evaluation receipts and goal constraints |
| Stop scheduling new work temporarily | Pause | Retained evidence and chronology remain available |
| Correct missing benchmark meaning | Contribution or coverage follow-up | Upstream policy, rubric, case, or coverage observation is named |

## Goal Contract and chronology

The Goal Contract is the session's authority boundary. Confirm it only after the target and measurement plan are resolvable. Later revisions should remain explicit rather than silently changing what earlier candidates attempted.

Chronology records durable session transitions, agent activity, proposed approaches, candidates, evaluations, pauses, resumptions, cancellation, and terminal state. Narrated trajectories expose bounded work segments and can identify unannounced durable activity without pretending to know private external-worker behavior.

## Candidates and current frontier

A candidate should identify its Harness version, hypothesis, parent or branch context, and evaluation evidence. The current frontier represents retained candidates supported by the latest valid evidence and Goal Contract. It is not synonymous with the most recent candidate or the highest aggregate score.

Evaluation receipts link candidates back to Benchmark Evaluations. If a candidate improves one objective but violates a constraint or regresses a must-level rubric, the frontier should keep that tradeoff visible.

## Human and external-worker boundary

Operators can pause, resume, cancel, review, or redirect work through the available session controls. External worker packages can carry the confirmed goal and evidence to Codex, Claude Code, or another worker. Teammately records only returned artifacts and requests that satisfy the return contract; it does not infer private work state.

{% example-demo title="Evolve a grounding harness" %}
The Goal Contract targets unsupported blends of current and superseded sources without regressing missing-source uncertainty. Evolve explores retrieval filtering, source ranking, and response-construction branches. Each candidate Harness version is evaluated against the pinned benchmark. The current frontier retains the candidates supported by both grounding and uncertainty evidence, while a newly discovered policy conflict becomes an Expert Contribution opportunity.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Related workflows" %}
- [Start an Improvement Session](/docs/improve/start-improvement-session)
- [Write and confirm a Goal Contract](/docs/improve/goal-contracts)
- [Choose Work or Evolve](/docs/improve/work-and-evolve)
- [Interpret candidates and the current frontier](/docs/improve/candidates-and-frontier)
- [Read chronology and trajectories](/docs/improve/chronology-and-trajectories)
- [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results)
- [Request an Expert Contribution](/docs/expert-contributions/request-contribution)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Coevolve](/docs/concepts/coevolve)
- [Harnesses](/docs/assets/harnesses)
- [Product boundaries](/docs/introduction/product-boundaries)
{% /related-card-grid %}

## Source confidence

Code-backed: the active versioned Improve workspace and session contract define starting evidence, Goal Contracts, Work and Evolve behavior, candidates, receipts, chronology, frontiers, and lifecycle controls. Internal coordinator implementation is supporting evidence, not a public API.
</doc>

<doc id="benchmark-evaluations.inspect" slug="benchmark-evaluations/inspect-results" url="https://teammately.ai/docs/benchmark-evaluations/inspect-results.md" title="Inspect Evaluation Results" updated="2026-09-13" status="stable">
---
id: benchmark-evaluations.inspect
title: Inspect Evaluation Results
summary: Trace Dashboard and List signals to Run, Case, Policy, Rubric, completeness, and telemetry evidence.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-09-13
canonical: /docs/benchmark-evaluations/inspect-results
---

# Inspect Evaluation Results

## Prerequisites

- A visible Run or Run Group with output or evaluation progress.
- Access to the exact Benchmark and Harness Version evidence.

Start with completeness and identity, then move from aggregate signals to the Cases and evaluator failures that support them.

## Steps

1. Open Dashboard and confirm the Benchmark Version, candidate Harness Version, Run Group type, attempt count, and evaluation progress.
2. Read rankings with their metric family and uncertainty. Distinguish average score, passed at least once, and passed every time over observed Runs. Inspect Run counts and coverage; historical group-specific pass@n and pass^n retain their original meanings.
3. Open **List → Runs** to inspect group and Run status, output progress, evaluation progress, metadata, and available resource telemetry.
4. Open **List → Evaluation results** for the Case summary, outcome, failed Policies, failed Rubrics, and evaluated count.
5. Use Arena for pairwise disagreement or Compare for a Harness matrix across Cases, evaluator facts, or Coverage Facets.
6. Classify the next action as candidate work, evaluator clarification, Case correction, coverage work, external-output remapping, or no action.

The List results surface is intentionally compact. Do not claim that it exposes full execution trajectories. The **Traces / Spans** segment currently reports a capability fence because the benchmark API does not provide evaluation execution traces.

## Reading incomplete and repeated evidence

An aggregate calculated over fewer evaluable Cases can look better while covering less evidence. Record evaluated, incomplete, and missing counts before comparing candidates. For repeated groups, inspect whether the configured number of attempts exists for every candidate and whether one failed attempt changes the metric interpretation.

Cost, tokens, and latency help route operational work but are nullable telemetry. Missing capture means unknown, not free or instantaneous execution.

> Evaluator authority
>
> Policy and Rubric results are the correctness evidence admitted by the Benchmark Version. Rankings and telemetry summarize that evidence; they do not create a new standard.

{% example-demo title="Example: apparent gain from incomplete evidence" %}
Harness B leads the overall table, but List shows that twelve difficult Cases are still unevaluated for B. Arena also reports incomplete pairs. The operator waits for terminal evidence instead of starting Improve from a ranking that covers a smaller Case population.
{% /example-demo %}

## Object and state changes

Inspection, filtering, and navigation are read-only. Starting Improve, a Contribution, coverage work, or a later Run creates separate durable work while preserving the inspected evidence.

## Success criteria

- Identity, completeness, metric family, and uncertainty are explicit.
- Important signals resolve to Cases and admitted evaluator outcomes.
- The next action targets the responsible artifact or candidate boundary.

## Common failure modes

- Reporting rank without the evaluated population.
- Inventing execution traces from the unavailable segment.
- Starting candidate work when the Case or Rubric is wrong.

## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Arena and Rankings](/docs/benchmark-evaluations/arena-and-rankings)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark results changed unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Benchmark runs](/docs/troubleshooting/benchmark-runs)
- [Missing outputs](/docs/troubleshooting/missing-outputs)
{% /related-card-grid %}

## Source confidence

Code-backed: Dashboard, List, Run detail, and workspace types establish result summaries, completion, rankings, repeated metrics, telemetry, and the current trace capability fence.
</doc>

<doc id="assets.harnesses" slug="assets/harnesses" url="https://teammately.ai/docs/assets/harnesses.md" title="Harnesses" updated="2026-08-22" status="stable">
---
id: assets.harnesses
title: Harnesses
summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
kind: reference
product_area: assets
status: stable
updated: 2026-08-22
canonical: /docs/assets/harnesses
---

# Harnesses

## Definition

A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.

A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.

## Fields, states, or lifecycle rules

- A newly created Harness begins with one blank editable Draft.
- Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded.
- The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
- Draft validation errors block publication. Warnings remain visible for operator review.
- **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also uses that Version for the currently selected benchmark.
- A previously saved Version can be selected with **Use this Version** for the current benchmark.
- Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
- Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
- Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
- Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
- Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`.
- Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation evidence.

## Draft, saved version, and activation

Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.

The Evaluations execution settings show project Harnesses with saved Versions and control which are active for future managed Runs. Activation is benchmark-scoped. Every admitted Run still records the exact saved Harness Version, so later activation changes do not rewrite existing evidence.

## Runtime, debug, and output mapping

Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.

Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness.

Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable.

{% example-demo title="Example: exact candidate identity" %}
An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work.
{% /example-demo %}

## Source confidence

Code-backed: the active Harness list and editor establish creation, archive state, Draft autosave and conflicts, validation, publication, activation, secrets, debug, Coding Agent proposals, and runtime preparation. The runnable adapter supports the execution boundary without making its internal API public.

## Related task pages

{% related-card-grid title="Related task pages" %}
- [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Start an Improvement Session](/docs/improve/start-improvement-session)
- [Configure Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
</doc>