# Submit a Managed Harness with the CLI
Generated: 2026-11-02T06:35:11.694Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: cli.managed-harness
title: Submit a Managed Harness with the CLI
summary: Select source files, submit a saved Harness, prepare its runtime and inspect every admitted execution.
kind: quickstart
product_area: developer_tools
status: stable
updated: 2026-10-31
canonical: /docs/cli/managed-harness
---
# Submit a Managed Harness with the CLI
## Prerequisites
Use authenticated write access to the selected Benchmark Version. Choose managed execution deliberately; [external output submission](/docs/cli/external-outputs) can keep execution in your own application runtime.
## Select source
Provide `main.py` with `generate_result(request)` for Python 3.12, and declare only selected files in `teammately-harness.json`:
### Example: Source selection manifest
```json
{"files":["main.py"],"dependency_specs":[]}
```
Read `teammately eval instructions --run "$BASELINE_RUN_ID"` when an actual baseline exists. Package at most 64 UTF-8 files, 256 KiB each and 1 MiB total. Symlinks, traversal, hidden/credential paths and cache/dependency directories are rejected. Server validation remains authoritative.
## Submit and prepare
```sh
teammately harness validate --source ./harness
teammately harness submit --source ./harness --name 'Application candidate' \
--receipt ./source-receipt.json
teammately harness prepare-runtime \
--harness "$HARNESS_ID" --harness-version "$HARNESS_VERSION_ID" \
--receipt ./runtime-receipt.json
teammately eval managed-profile
```
Take Harness IDs from submission and inspect preparation state. Its acknowledgement does not certify readiness. Read the current server profile and retain its concurrency token.
## Request execution
```sh
teammately eval run \
--harness "$HARNESS_ID" --harness-version "$HARNESS_VERSION_ID" \
--profile-revision "$PROFILE_TOKEN" --receipt ./admission-receipt.json
```
`PROFILE_TOKEN` is the exact server-returned value, passed under the existing flag name. Do not invent it. Use a separate receipt for each stage. Managed policy can admit multiple executions; inspect every returned child Run ID with `run get`, `run wait` and `eval results`.
## Recovery and success
Replay an interrupted stage with the original inputs and its existing receipt. A changed source package is a different operation. Keep saved source, runtime readiness, admission and completed judging separate. Submission does not publish or activate the Harness.
## Related reference pages
{% related-card-grid %}
- [Managed execution API](/docs/public-api/managed-execution)
- [Command reference](/docs/cli/reference)
- [Benchmark Run troubleshooting](/docs/troubleshooting/benchmark-runs)
{% /related-card-grid %}
---
id: cli.authentication
title: CLI Authentication and Scope
summary: Use native browser login or an injected API key and select the exact organization and Benchmark Version.
kind: reference
product_area: developer_tools
status: stable
updated: 2026-10-31
canonical: /docs/cli/authentication
---
# CLI Authentication and Scope
## Definition
Use native browser login for interactive External Agents work. Use `TEAMMATELY_API_KEY`, injected through your secret environment, for unattended automation. An environment API key takes precedence over a saved login.
## Sign in and inspect access
```sh
teammately login --profile work
teammately status
teammately switch --profile work
```
Login opens browser authorization with PKCE and a temporary loopback callback, then selects an organization. `--organization` is an optional shortcut. Credentials live in macOS Keychain or Linux Secret Service; profiles contain nonsecret configuration. Use a distinct profile for another organization/account/API origin.
`status` refreshes and checks organization access against the API. It does not establish permission to every Project. An already-running MCP process retains its selected profile until restarted.
## Fields, states, or lifecycle rules
| Variable | Meaning |
| --- | --- |
| `TEAMMATELY_API_KEY` | Secret API-key override supplied by the host |
| `TEAMMATELY_PROFILE` | Selected native-login profile |
| `TEAMMATELY_ORGANIZATION` | Organization key |
| `TEAMMATELY_PROJECT` | Issued Project ID |
| `TEAMMATELY_BENCHMARK` | Issued Benchmark ID |
| `TEAMMATELY_BENCHMARK_VERSION` | Exact immutable Version ID |
| `TEAMMATELY_API_BASE` | Trusted API base, default `https://api.teammately.ai/v1` |
CLI scope flags `--organization`, `--project`, `--benchmark`, `--version` override environment defaults. MCP calls supply explicit scope each time. Never choose the API origin from Case text or model output.
Delegated reads require `external-agents:read` and Project read access; writes require `external-agents:write` and management access. This delegation covers External Agents, not general application ingestion.
## Sign out and recover
```sh
teammately logout --profile work
```
Logout requires network access to revoke the refresh token before deleting the local credential. Other profiles, API keys and the browser session are separate. Issued access tokens retain their expiry.
For `401`, inspect the active credential and login status. For `403`, inspect organization, Project, action and key restrictions. A Fieldloop-only key is not an External Agents credential. Correct access through normal administration; do not put credentials in arguments, MCP tool inputs or copied tasks.
## Related task pages
{% related-card-grid %}
- [MCP setup](/docs/cli/mcp)
- [External output submission](/docs/cli/external-outputs)
- [Authentication troubleshooting](/docs/troubleshooting/authentication)
{% /related-card-grid %}
---
id: public-api.managed-execution
title: Managed Harness Execution API
summary: Submit Harness source, prepare its runtime and request execution as distinct recoverable operations.
kind: api
product_area: public_api
status: stable
updated: 2026-10-31
canonical: /docs/public-api/managed-execution
---
# Managed Harness Execution API
## Definition
Managed execution runs an explicitly submitted Harness against the selected Benchmark Version. It is an alternative to [submitting application outputs](/docs/public-api/evaluation-outputs). Paths below are relative to `BENCHMARK_URL` from the [quickstart](/docs/public-api/quickstart).
## Operation sequence
| Method and path | Input | Retain |
| --- | --- | --- |
| `GET /managed-run-profile` | Selected scope | Execution policy and concurrency token |
| `POST /harness-submissions` | `name`, exact `source` | Issued Harness/Version IDs |
| `POST /harness-runtime-preparations` | `harness_id`, `harness_version_id` | Preparation acknowledgement/state |
| `POST /managed-runs` | Saved IDs and `expected_managed_run_profile_revision` | Every admitted execution |
Writes require the operation's `Idempotency-Key`. Preserve exact bytes and a separate key for each command.
## Source package
`source.schema_version` is `harness_working_copy.v1`. File entries have relative `path` and UTF-8 `content_text`. The runtime accepts Python `3.12` and the callable `main.py:generate_result(request)`. Source policy permits at most 64 files, 256 KiB each and 1 MiB total. Symlinks, traversal and credential/cache paths are rejected.
Use the [CLI managed-Harness guide](/docs/cli/managed-harness) to package local files and retain recovery receipts. The [OpenAPI](https://api.teammately.ai/v1/openapi.yaml) defines all source/runtime fields.
## Prepare and admit
Preparation uses the exact saved IDs; acknowledgement does not certify runtime readiness. Inspect returned state and failures. Admission takes those IDs and the current `expected_managed_run_profile_revision`, plus optional existing `compute_profile_id`. The profile token is a concurrency precondition supplied by the server, not a number to choose.
Managed admission can return multiple individual executions according to policy. Inspect every returned ID through the [results API](/docs/public-api/benchmark-results). Admission is not completed generation/judging, and it does not activate a Harness.
Prepare `managed-run.json` with the saved `harness_id` and exact `harness_version_id`. Read `GET /managed-run-profile` and copy its integer `data.revision` into the request field `expected_managed_run_profile_revision`. Export `BENCHMARK_URL` and `MANAGED_OPERATION_KEY`; the latter identifies this logical admission command. These examples submit the file bytes once, without adding automatic retries.
### Example: Request managed execution
```sh
curl --fail-with-body \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
--header 'Content-Type: application/json' \
--header "Idempotency-Key: ${MANAGED_OPERATION_KEY}" \
--data-binary @managed-run.json \
"${BENCHMARK_URL}/managed-runs"
```
```python
import os
from pathlib import Path
from urllib.parse import quote
from urllib.request import HTTPRedirectHandler, Request, build_opener
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]
url = os.environ["BENCHMARK_URL"] + "/managed-runs"
headers = {"Authorization": "Bearer " + api_key}
headers["Idempotency-Key"] = os.environ["MANAGED_OPERATION_KEY"]
headers["Content-Type"] = "application/json"
body = Path("managed-run.json").read_bytes()
request = Request(url, data=body, headers=headers, method="POST")
with opener.open(request) as response:
print(response.read().decode("utf-8"))
```
```typescript
import { readFile } from "node:fs/promises";
const apiKey = process.env.TEAMMATELY_API_KEY!;
{
const url = new URL(process.env.BENCHMARK_URL! + "/managed-runs");
const response = await fetch(url, {
method: "POST",
redirect: "error",
headers: {
Authorization: `Bearer ${apiKey}`,
"Idempotency-Key": process.env.MANAGED_OPERATION_KEY!,
"Content-Type": "application/json",
},
body: new Uint8Array(await readFile("managed-run.json")).buffer,
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
console.log(await response.text());
}
```
```go
package main
import (
"bytes"
"context"
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
if err := run(context.Background()); err != nil { log.Fatal(err) }
}
func run(ctx context.Context) error {
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
{
target := os.Getenv("BENCHMARK_URL") + "/managed-runs"
payload, err := os.ReadFile("managed-run.json")
if err != nil { return err }
request, err := http.NewRequestWithContext(ctx, "POST", target, bytes.NewReader(payload))
if err != nil { return err }
request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
request.Header.Set("Idempotency-Key", os.Getenv("MANAGED_OPERATION_KEY"))
request.Header.Set("Content-Type", "application/json")
response, err := client.Do(request)
if err != nil { return err }
payloadBytes, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil { return readErr }
fmt.Println(string(payloadBytes))
if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
}
return nil
}
```
## Related reference pages
{% related-card-grid %}
- [Authentication](/docs/public-api/authentication)
- [CLI managed Harness](/docs/cli/managed-harness)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}
---
id: assets.harnesses
title: Harnesses
summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
kind: reference
product_area: assets
status: stable
updated: 2026-10-30
canonical: /docs/assets/harnesses
---
# Harnesses
## Definition
A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.
A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.
## Fields, states, or lifecycle rules
- A newly created Harness begins with one blank editable Draft.
- Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded.
- The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
- Draft validation errors block publication. Warnings remain visible for operator review.
- **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also requests use of that Version for the selected Benchmark; confirm its **In use** state.
- A previously saved Version can be selected with **Use this Version** for the current benchmark.
- Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
- Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
- Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
- Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
- Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`.
- Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation results.
## Draft, saved version, and activation
Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.
The Harness editor owns **Publish** and **Use this Version**. Those actions select a saved Version for the current Benchmark. The evaluation launch chooses saved Versions, machine and Trials; it does not provide the retired activation-settings control. Every admitted Run keeps its exact saved Version, so a later activation change does not rewrite earlier results.
## Runtime, debug, and output mapping
Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.
Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness.
Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable.
{% example-demo title="Example: exact candidate identity" %}
An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Start a Lemon Goal Session](/docs/improve/start-improvement-session)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}