# Managed Harness Execution API

Generated: 2026-11-02T05:43:03.607Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="public-api.managed-execution" slug="public-api/managed-execution" url="https://teammately.ai/docs/public-api/managed-execution.md" title="Managed Harness Execution API" updated="2026-10-31" status="stable">
---
id: public-api.managed-execution
title: Managed Harness Execution API
summary: Submit Harness source, prepare its runtime and request execution as distinct recoverable operations.
kind: api
product_area: public_api
status: stable
updated: 2026-10-31
canonical: /docs/public-api/managed-execution
---

# Managed Harness Execution API

## Definition

Managed execution runs an explicitly submitted Harness against the selected Benchmark Version. It is an alternative to [submitting application outputs](/docs/public-api/evaluation-outputs). Paths below are relative to `BENCHMARK_URL` from the [quickstart](/docs/public-api/quickstart).

## Operation sequence

| Method and path | Input | Retain |
| --- | --- | --- |
| `GET /managed-run-profile` | Selected scope | Execution policy and concurrency token |
| `POST /harness-submissions` | `name`, exact `source` | Issued Harness/Version IDs |
| `POST /harness-runtime-preparations` | `harness_id`, `harness_version_id` | Preparation acknowledgement/state |
| `POST /managed-runs` | Saved IDs and `expected_managed_run_profile_revision` | Every admitted execution |

Writes require the operation's `Idempotency-Key`. Preserve exact bytes and a separate key for each command.

## Source package

`source.schema_version` is `harness_working_copy.v1`. File entries have relative `path` and UTF-8 `content_text`. The runtime accepts Python `3.12` and the callable `main.py:generate_result(request)`. Source policy permits at most 64 files, 256 KiB each and 1 MiB total. Symlinks, traversal and credential/cache paths are rejected.

Use the [CLI managed-Harness guide](/docs/cli/managed-harness) to package local files and retain recovery receipts. The [OpenAPI](https://api.teammately.ai/v1/openapi.yaml) defines all source/runtime fields.

## Prepare and admit

Preparation uses the exact saved IDs; acknowledgement does not certify runtime readiness. Inspect returned state and failures. Admission takes those IDs and the current `expected_managed_run_profile_revision`, plus optional existing `compute_profile_id`. The profile token is a concurrency precondition supplied by the server, not a number to choose.

Managed admission can return multiple individual executions according to policy. Inspect every returned ID through the [results API](/docs/public-api/benchmark-results). Admission is not completed generation/judging, and it does not activate a Harness.

Prepare `managed-run.json` with the saved `harness_id` and exact `harness_version_id`. Read `GET /managed-run-profile` and copy its integer `data.revision` into the request field `expected_managed_run_profile_revision`. Export `BENCHMARK_URL` and `MANAGED_OPERATION_KEY`; the latter identifies this logical admission command. These examples submit the file bytes once, without adding automatic retries.

### Example: Request managed execution

```sh
curl --fail-with-body \
  --header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
  --header 'Content-Type: application/json' \
  --header "Idempotency-Key: ${MANAGED_OPERATION_KEY}" \
  --data-binary @managed-run.json \
  "${BENCHMARK_URL}/managed-runs"
```

```python
import os
from pathlib import Path
from urllib.parse import quote
from urllib.request import HTTPRedirectHandler, Request, build_opener

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]

url = os.environ["BENCHMARK_URL"] + "/managed-runs"
headers = {"Authorization": "Bearer " + api_key}
headers["Idempotency-Key"] = os.environ["MANAGED_OPERATION_KEY"]
headers["Content-Type"] = "application/json"
body = Path("managed-run.json").read_bytes()
request = Request(url, data=body, headers=headers, method="POST")
with opener.open(request) as response:
    print(response.read().decode("utf-8"))
```

```typescript
import { readFile } from "node:fs/promises";

const apiKey = process.env.TEAMMATELY_API_KEY!;

{
  const url = new URL(process.env.BENCHMARK_URL! + "/managed-runs");
  const response = await fetch(url, {
    method: "POST",
    redirect: "error",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Idempotency-Key": process.env.MANAGED_OPERATION_KEY!,
      "Content-Type": "application/json",
    },
    body: new Uint8Array(await readFile("managed-run.json")).buffer,
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  console.log(await response.text());
}
```

```go
package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
)

func main() {
    if err := run(context.Background()); err != nil { log.Fatal(err) }
}

func run(ctx context.Context) error {
    client := &http.Client{
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    {
        target := os.Getenv("BENCHMARK_URL") + "/managed-runs"
        payload, err := os.ReadFile("managed-run.json")
        if err != nil { return err }
        request, err := http.NewRequestWithContext(ctx, "POST", target, bytes.NewReader(payload))
        if err != nil { return err }
        request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
        request.Header.Set("Idempotency-Key", os.Getenv("MANAGED_OPERATION_KEY"))
        request.Header.Set("Content-Type", "application/json")
        response, err := client.Do(request)
        if err != nil { return err }
        payloadBytes, readErr := io.ReadAll(response.Body)
        response.Body.Close()
        if readErr != nil { return readErr }
        fmt.Println(string(payloadBytes))
        if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
    }
    return nil
}
```

## Related reference pages

{% related-card-grid %}
- [Authentication](/docs/public-api/authentication)
- [CLI managed Harness](/docs/cli/managed-harness)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}
</doc>

<doc id="public-api.overview" slug="public-api" url="https://teammately.ai/docs/public-api.md" title="Public API" updated="2026-10-31" status="stable">
---
id: public-api.overview
title: Public API
summary: Connect applications to Cases, Fieldloop ingestion, benchmark execution and evaluation results through the organization-scoped API.
kind: api
product_area: public_api
status: stable
updated: 2026-10-31
canonical: /docs/public-api
---

# Public API

## Definition

The customer gateway uses the base `https://api.teammately.ai/v1` and organization-scoped resources beneath `/organizations/{organizationKey}/projects/{projectId}`.

Start with [Your First API Request](/docs/public-api/quickstart). Use REST for direct request control, [SDKs](/docs/sdks) for application ingestion, or the [CLI](/docs/cli) for coding-agent and local-file workflows.

## Contract families

The [endpoint catalog](/docs/public-api/endpoints) includes Projects, dimensions/terms and dataset operations as well as the workflow families below.

| Family | Operations and guide |
| --- | --- |
| [Cases](/docs/public-api/cases) | List/detail and single/batch reference responses |
| [Case imports](/docs/public-api/case-imports) | Multipart import and status |
| [Fieldloop](/docs/public-api/fieldloop) | Batch submission and receipts |
| [External outputs](/docs/public-api/evaluation-outputs) | Frozen manifest, real first results, batches and completion |
| [Managed execution](/docs/public-api/managed-execution) | Source submission, runtime preparation and admission |
| [Results](/docs/public-api/benchmark-results) | Pinned context, individual Trial results and comparisons |

## OpenAPI and conventions

The [OpenAPI YAML](https://api.teammately.ai/v1/openapi.yaml) and [OpenAPI JSON](https://api.teammately.ai/v1/openapi.json) describe request fields, response schemas and per-operation authentication. These guides explain operation sequences and asynchronous outcomes.

Read [Authentication](/docs/public-api/authentication) and [Conventions](/docs/public-api/conventions) before writing. Envelopes and pagination vary by resource. Supply credentials through the host's secret mechanism and use identifiers from Teammately.

Use the request panel's language selector to switch between equivalent cURL, Python, TypeScript and Go examples. Copy copies the selected language. JSON request bodies and HTTP signatures are shown separately.

These are direct HTTP examples, not SDK method declarations. Python uses its standard library; TypeScript uses Node.js 22+ built-in fetch and file APIs; Go uses its standard library. [Application SDKs](/docs/sdks) describe the resources each Teammately library exposes.

The [quickstart](/docs/public-api/quickstart) exports the nonsecret `PROJECT_URL` and `BENCHMARK_URL` values. Export each example's additional identifiers and logical-command keys before running it. Supply `TEAMMATELY_API_KEY` through your approved secret environment. Keep original body files and request identity when following the operation's recovery rules.

## Runs, Trials and wire names

The product calls an external cohort a **Run** and child executions **Trials**. API paths retain `run-groups` for the cohort and `runs` for an individual child. Keep both returned IDs. Assessment and comparison reads use the child `runId`, not the group ID.
## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [API Authentication](/docs/public-api/authentication)
- [API Conventions](/docs/public-api/conventions)
- [Cases API](/docs/public-api/cases)
- [Import Cases](/docs/public-api/case-imports)
- [Submit External Evaluation Outputs](/docs/public-api/evaluation-outputs)
- [Retrieve Benchmark Results](/docs/public-api/benchmark-results)
- [Result Serialization](/docs/public-api/result-schema)
{% /related-card-grid %}
</doc>

<doc id="public-api.benchmark-results" slug="public-api/benchmark-results" url="https://teammately.ai/docs/public-api/benchmark-results.md" title="Retrieve Benchmark Results" updated="2026-10-31" status="stable">
---
id: public-api.benchmark-results
title: Retrieve Benchmark Results
summary: Read scoped individual Run context, complete rubric results, instructions, and comparable Run differences.
kind: api
product_area: public_api
status: stable
updated: 2026-10-31
canonical: /docs/public-api/benchmark-results
---

# Retrieve Benchmark Results

## Definition

Under the organization/project/Benchmark/Version prefix, current gateway routes include:

```http
GET /version-context
GET /runs/{runId}
GET /runs/{runId}/evaluation-context
GET /runs/{runId}/evaluation-results
GET /runs/{runId}/evaluation-instructions
GET /comparison/{runId}/summary
GET /comparison/{runId}/cases
```

External Group/child reads additionally use their returned nested links.

## Read the exact population

Verify that the individual Run belongs to the selected Benchmark Version. Read available Case outputs, rubric judgments, native interpretation, rationale and incomplete state. Page through the authoritative filtered result population rather than treating the first page as all results.

Instructions and context orient the coding agent; neither is a mutation or a completed evaluation.

## Compare

Comparisons preserve individual Run grain and recorded conditions. Missing outputs, profile drift or unavailable native results can prevent a definitive comparison. A later independent Run does not silently replace the reference chosen before outcomes were known.

Record the exact starting and candidate Run IDs. Do not interpret legacy compatibility columns as mandatory binary rules when the current Regime supplies native point/gate meaning.

## Read a bounded page

### Example: Read evaluation results

```sh
curl --fail-with-body \
  --header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
  "${BENCHMARK_URL}/runs/${RUN_ID}/evaluation-results?limit=20&offset=0"
```

```python
import os
from urllib.parse import quote, urlencode
from urllib.request import HTTPRedirectHandler, Request, build_opener

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]

url = os.environ["BENCHMARK_URL"] + "/runs/" + quote(os.environ["RUN_ID"], safe="") + "/evaluation-results"
url += "?" + urlencode({"limit": "20", "offset": "0"})
headers = {"Authorization": "Bearer " + api_key}
request = Request(url, headers=headers, method="GET")
with opener.open(request) as response:
    print(response.read().decode("utf-8"))
```

```typescript
const apiKey = process.env.TEAMMATELY_API_KEY!;

{
  const url = new URL(process.env.BENCHMARK_URL! + "/runs/" + encodeURIComponent(process.env.RUN_ID!) + "/evaluation-results");
  url.searchParams.set("limit", "20");
  url.searchParams.set("offset", "0");
  const response = await fetch(url, {
    method: "GET",
    redirect: "error",
    headers: {
      Authorization: `Bearer ${apiKey}`,
    },
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  console.log(await response.text());
}
```

```go
package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/url"
    "os"
)

func main() {
    if err := run(context.Background()); err != nil { log.Fatal(err) }
}

func run(ctx context.Context) error {
    client := &http.Client{
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    {
        target := os.Getenv("BENCHMARK_URL") + "/runs/" + url.PathEscape(os.Getenv("RUN_ID")) + "/evaluation-results"
        query := url.Values{}
        query.Set("limit", "20")
        query.Set("offset", "0")
        target += "?" + query.Encode()
        request, err := http.NewRequestWithContext(ctx, "GET", target, nil)
        if err != nil { return err }
        request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
        response, err := client.Do(request)
        if err != nil { return err }
        payloadBytes, readErr := io.ReadAll(response.Body)
        response.Body.Close()
        if readErr != nil { return readErr }
        fmt.Println(string(payloadBytes))
        if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
    }
    return nil
}
```

Use the child Trial's `runId`. The `data` payload holds the page. `limit` accepts 1–100 (default 20); `offset` is nonnegative. Page through the returned population instead of treating one page as all results.

| Query | Selection |
| --- | --- |
| `result` | `passed`, `failed` or `missing` rubric observations |
| `case_id`, `policy_id` | One Case or Policy |
| `case_rubric_pair_id` | One Case/Rubric pair |
| `dimension_id`, `ontology_id` | Declared coverage population |

The same filters apply to `evaluation-context`. Unknown/repeated parameters are rejected. Run and instruction reads take no query parameters.

## Compare two Trials

### Example: Compare two Trials

```sh
curl --fail-with-body --get \
  --header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
  --data-urlencode "target_run_id=${TARGET_RUN_ID}" \
  "${BENCHMARK_URL}/comparison/${BASELINE_RUN_ID}/summary"
```

```python
import os
from urllib.parse import quote, urlencode
from urllib.request import HTTPRedirectHandler, Request, build_opener

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]

url = os.environ["BENCHMARK_URL"] + "/comparison/" + quote(os.environ["BASELINE_RUN_ID"], safe="") + "/summary"
url += "?" + urlencode({"target_run_id": os.environ["TARGET_RUN_ID"]})
headers = {"Authorization": "Bearer " + api_key}
request = Request(url, headers=headers, method="GET")
with opener.open(request) as response:
    print(response.read().decode("utf-8"))
```

```typescript
const apiKey = process.env.TEAMMATELY_API_KEY!;

{
  const url = new URL(process.env.BENCHMARK_URL! + "/comparison/" + encodeURIComponent(process.env.BASELINE_RUN_ID!) + "/summary");
  url.searchParams.set("target_run_id", process.env.TARGET_RUN_ID!);
  const response = await fetch(url, {
    method: "GET",
    redirect: "error",
    headers: {
      Authorization: `Bearer ${apiKey}`,
    },
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  console.log(await response.text());
}
```

```go
package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/url"
    "os"
)

func main() {
    if err := run(context.Background()); err != nil { log.Fatal(err) }
}

func run(ctx context.Context) error {
    client := &http.Client{
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    {
        target := os.Getenv("BENCHMARK_URL") + "/comparison/" + url.PathEscape(os.Getenv("BASELINE_RUN_ID")) + "/summary"
        query := url.Values{}
        query.Set("target_run_id", os.Getenv("TARGET_RUN_ID"))
        target += "?" + query.Encode()
        request, err := http.NewRequestWithContext(ctx, "GET", target, nil)
        if err != nil { return err }
        request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
        response, err := client.Do(request)
        if err != nil { return err }
        payloadBytes, readErr := io.ReadAll(response.Body)
        response.Body.Close()
        if readErr != nil { return readErr }
        fmt.Println(string(payloadBytes))
        if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
    }
    return nil
}
```

Use `/cases` for paged changes, with `limit` and `offset`. Both child IDs must belong to the scope. Inspect comparability and incompleteness before interpreting a delta.
## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Result Serialization](/docs/public-api/result-schema)
- [Submit External Evaluation Outputs](/docs/public-api/evaluation-outputs)
- [Evaluation Results](/docs/benchmark-evaluations/results)
{% /related-card-grid %}
</doc>

<doc id="assets.harnesses" slug="assets/harnesses" url="https://teammately.ai/docs/assets/harnesses.md" title="Harnesses" updated="2026-10-30" status="stable">
---
id: assets.harnesses
title: Harnesses
summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
kind: reference
product_area: assets
status: stable
updated: 2026-10-30
canonical: /docs/assets/harnesses
---

# Harnesses

## Definition

A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.

A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.

## Fields, states, or lifecycle rules

- A newly created Harness begins with one blank editable Draft.
- Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded.
- The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
- Draft validation errors block publication. Warnings remain visible for operator review.
- **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also requests use of that Version for the selected Benchmark; confirm its **In use** state.
- A previously saved Version can be selected with **Use this Version** for the current benchmark.
- Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
- Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
- Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
- Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
- Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`.
- Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation results.

## Draft, saved version, and activation

Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.

The Harness editor owns **Publish** and **Use this Version**. Those actions select a saved Version for the current Benchmark. The evaluation launch chooses saved Versions, machine and Trials; it does not provide the retired activation-settings control. Every admitted Run keeps its exact saved Version, so a later activation change does not rewrite earlier results.

## Runtime, debug, and output mapping

Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.

Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness.

Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable.

{% example-demo title="Example: exact candidate identity" %}
An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work.
{% /example-demo %}

## Related task pages

{% related-card-grid title="Related task pages" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Start a Lemon Goal Session](/docs/improve/start-improvement-session)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
</doc>