# API Conventions Generated: 2026-11-02T06:36:19.723Z Source build: local Canonical docs: https://teammately.ai/docs --- id: public-api.conventions title: API Conventions summary: Use organization-scoped paths, operation-specific envelopes, issued IDs, exact retry identity, and explicit completion. kind: api product_area: public_api status: stable updated: 2026-10-31 canonical: /docs/public-api/conventions --- # API Conventions ## Definition The resource prefix is `/v1/organizations/{organizationKey}/projects/{projectId}`. Benchmark operations continue beneath the selected Benchmark and immutable Version. Paths and IDs are scoped; a readable title is not a join key. ## Envelopes and pagination Case lists return `message` and `data: {columns, rows}`; detail returns `message` and `result`. Fieldloop and benchmark operations use their documented `data` payloads. Parse the called operation's envelope. Case reads use limit/offset pagination, defaulting to 10 rows and accepting a positive limit through 1,000. Other readers have their own pagination. Preserve returned continuation information instead of assuming a cursor shape universally. Case reads emit ETag and support If-None-Match. Concurrency and mutation preconditions belong to the particular operation. ## Writes and retries Use an Idempotency-Key where the operation requires it. Preserve exact request content and logical identity, but follow that operation's recovery contract. Changed content is a different command. External results append only to the issued Trial. Completion freezes accepted responses and admits evaluation separately. A failed admission retry does not replace those responses. ## IDs and errors Use database/API-issued opaque IDs. Example placeholders are not usable IDs. Keep the original HTTP status, structured code and actionable reason; no global error envelope is promised across legacy and forwarded families. ## HTTP outcomes | Outcome | Next step | | --- | --- | | `200` or `201` | Inspect the resource or per-item outcomes | | `202` | Retain the issued identity and inspect processing state | | `304` | Use the representation associated with the matching ETag | | `400` or `422` | Correct the request using its code and reason | | `401` or `403` | Correct credentials or scope | | `409` or `412` | Inspect immutable state, idempotency or preconditions | | `429` | Honor `Retry-After` when supplied | | `502`, `503` or `504` | Preserve command identity before recovering an uncertain write | Not every endpoint emits every status. The [OpenAPI](https://api.teammately.ai/v1/openapi.yaml) lists operation-specific responses; one rate-limit response does not define a universal quota. ## Recover without duplicating work Fieldloop and benchmark submission workflows support their documented exact-content recovery. Preserve original bytes/key and inspect the returned receipt. A client timeout does not undo admitted work. Case imports and batch Case reference writes use duplicate-key rejection: repeating a key returns `409`, not a replay of the original response. Retain the issued import identity when available and read its status. If a response is lost before its identity is known, reconcile through your administrator or support before submitting with a new key; a fresh key can create another operation. Do not assume duplicate rejection proves whether the first handler completed. See [SDK Delivery and Recovery](/docs/sdks/delivery-and-recovery) for the Fieldloop client path. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Public API](/docs/public-api) - [API Authentication](/docs/public-api/authentication) - [Submit External Evaluation Outputs](/docs/public-api/evaluation-outputs) {% /related-card-grid %} --- id: public-api.overview title: Public API summary: Connect applications to Cases, Fieldloop ingestion, benchmark execution and evaluation results through the organization-scoped API. kind: api product_area: public_api status: stable updated: 2026-10-31 canonical: /docs/public-api --- # Public API ## Definition The customer gateway uses the base `https://api.teammately.ai/v1` and organization-scoped resources beneath `/organizations/{organizationKey}/projects/{projectId}`. Start with [Your First API Request](/docs/public-api/quickstart). Use REST for direct request control, [SDKs](/docs/sdks) for application ingestion, or the [CLI](/docs/cli) for coding-agent and local-file workflows. ## Contract families The [endpoint catalog](/docs/public-api/endpoints) includes Projects, dimensions/terms and dataset operations as well as the workflow families below. | Family | Operations and guide | | --- | --- | | [Cases](/docs/public-api/cases) | List/detail and single/batch reference responses | | [Case imports](/docs/public-api/case-imports) | Multipart import and status | | [Fieldloop](/docs/public-api/fieldloop) | Batch submission and receipts | | [External outputs](/docs/public-api/evaluation-outputs) | Frozen manifest, real first results, batches and completion | | [Managed execution](/docs/public-api/managed-execution) | Source submission, runtime preparation and admission | | [Results](/docs/public-api/benchmark-results) | Pinned context, individual Trial results and comparisons | ## OpenAPI and conventions The [OpenAPI YAML](https://api.teammately.ai/v1/openapi.yaml) and [OpenAPI JSON](https://api.teammately.ai/v1/openapi.json) describe request fields, response schemas and per-operation authentication. These guides explain operation sequences and asynchronous outcomes. Read [Authentication](/docs/public-api/authentication) and [Conventions](/docs/public-api/conventions) before writing. Envelopes and pagination vary by resource. Supply credentials through the host's secret mechanism and use identifiers from Teammately. Use the request panel's language selector to switch between equivalent cURL, Python, TypeScript and Go examples. Copy copies the selected language. JSON request bodies and HTTP signatures are shown separately. These are direct HTTP examples, not SDK method declarations. Python uses its standard library; TypeScript uses Node.js 22+ built-in fetch and file APIs; Go uses its standard library. [Application SDKs](/docs/sdks) describe the resources each Teammately library exposes. The [quickstart](/docs/public-api/quickstart) exports the nonsecret `PROJECT_URL` and `BENCHMARK_URL` values. Export each example's additional identifiers and logical-command keys before running it. Supply `TEAMMATELY_API_KEY` through your approved secret environment. Keep original body files and request identity when following the operation's recovery rules. ## Runs, Trials and wire names The product calls an external cohort a **Run** and child executions **Trials**. API paths retain `run-groups` for the cohort and `runs` for an individual child. Keep both returned IDs. Assessment and comparison reads use the child `runId`, not the group ID. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [API Authentication](/docs/public-api/authentication) - [API Conventions](/docs/public-api/conventions) - [Cases API](/docs/public-api/cases) - [Import Cases](/docs/public-api/case-imports) - [Submit External Evaluation Outputs](/docs/public-api/evaluation-outputs) - [Retrieve Benchmark Results](/docs/public-api/benchmark-results) - [Result Serialization](/docs/public-api/result-schema) {% /related-card-grid %} --- id: public-api.authentication title: API Authentication summary: Distinguish API-key capability, delegated External Agents login, current destination access, and browser authentication. kind: api product_area: public_api status: stable updated: 2026-10-31 canonical: /docs/public-api/authentication --- # API Authentication ## Definition The gateway accepts API keys and delegated native-login access tokens on their supported routes. Browser session cookies remain the interactive app path; these credential types are not interchangeable. ## API keys Keys belong to a workspace. Base capability can be read-only, full access or Fieldloop submission. Project restrictions, allowed actions and expiry narrow that capability when supplied. A new Fieldloop submission key carries that workspace capability. The caller still supplies the Project and source destination, and current access is checked at intake. Previously issued restricted keys retain their restrictions. Changing access requires issuing the intended credential; renaming a key does not broaden it. Never place values in URLs, source, logs or copied agent tasks. ## Delegated login Native login is scoped to supported External Agents benchmark routes and organization login status. Delegation uses `external-agents:read` and `external-agents:write`, plus current project read/manage authorization. It does not provision a user, select a fallback organization or grant every endpoint. The native client uses authorization-code login with PKCE and an OS credential store. Auth discovery/session routes are distinct from customer operations. ## Request headers Supply the credential as a bearer token. `TEAMMATELY_API_KEY` is injected by your secret mechanism; configure `PROJECT_URL` with the [quickstart](/docs/public-api/quickstart). ### Example: Read a Case page ```sh curl --fail-with-body \ --header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \ "${PROJECT_URL}/cases?limit=1&offset=0" ``` ```python import os from urllib.parse import quote, urlencode from urllib.request import HTTPRedirectHandler, Request, build_opener class NoRedirect(HTTPRedirectHandler): def redirect_request(self, req, fp, code, msg, headers, newurl): return None opener = build_opener(NoRedirect()) api_key = os.environ["TEAMMATELY_API_KEY"] url = os.environ["PROJECT_URL"] + "/cases" url += "?" + urlencode({"limit": "1", "offset": "0"}) headers = {"Authorization": "Bearer " + api_key} request = Request(url, headers=headers, method="GET") with opener.open(request) as response: print(response.read().decode("utf-8")) ``` ```typescript const apiKey = process.env.TEAMMATELY_API_KEY!; { const url = new URL(process.env.PROJECT_URL! + "/cases"); url.searchParams.set("limit", "1"); url.searchParams.set("offset", "0"); const response = await fetch(url, { method: "GET", redirect: "error", headers: { Authorization: `Bearer ${apiKey}`, }, }); if (!response.ok) throw new Error(`HTTP ${response.status}`); console.log(await response.text()); } ``` ```go package main import ( "context" "fmt" "io" "log" "net/http" "net/url" "os" ) func main() { if err := run(context.Background()); err != nil { log.Fatal(err) } } func run(ctx context.Context) error { client := &http.Client{ CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } { target := os.Getenv("PROJECT_URL") + "/cases" query := url.Values{} query.Set("limit", "1") query.Set("offset", "0") target += "?" + query.Encode() request, err := http.NewRequestWithContext(ctx, "GET", target, nil) if err != nil { return err } request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY")) response, err := client.Do(request) if err != nil { return err } payloadBytes, readErr := io.ReadAll(response.Body) response.Body.Close() if readErr != nil { return readErr } fmt.Println(string(payloadBytes)) if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) } } return nil } ``` ## Choose the credential | Operation | Credential boundary | | --- | --- | | Case reads | API key with Project read access | | Case imports/reference writes | API key with the required write access | | Fieldloop submission/receipts | Fieldloop capability, destination permission and original-submitter receipt ownership | | External Agents reads | Authorized API key, or delegated `external-agents:read` plus Project read access | | External Agents writes | Authorized API key, or delegated `external-agents:write` plus Project management access | For delegated terminal access, follow [CLI Authentication](/docs/cli/authentication). Application ingestion uses an API key without browser login. A read-only MCP tool does not narrow the credential's permission elsewhere. ## Access failures For `401`, check the credential's presence, validity and API origin. For `403`, check the workspace, Project, capability, restrictions and delegated scope. Organization IP restrictions can apply. Correct access through normal administration. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Public API](/docs/public-api) - [API Conventions](/docs/public-api/conventions) - [API Keys and Delegated Access](/docs/admin-console/api-keys) {% /related-card-grid %} --- id: public-api.evaluation-outputs title: Submit External Evaluation Outputs summary: Create an external Run from actual first results, append missing Case responses, and explicitly complete its Trial for evaluation. kind: api product_area: public_api status: stable updated: 2026-10-31 canonical: /docs/public-api/evaluation-outputs --- # Submit External Evaluation Outputs ## Definition Use the organization/project/Benchmark/Version prefix and retrieve `GET /case-manifest` for exact frozen Cases. Product Runs and Trials retain the API's existing `run-groups` and child `runs` wire identities. ## Create with actual results ```http POST /run-groups Idempotency-Key: ``` ```json { "name": "Application comparison", "results": [ { "case_id": "", "output": "" } ] } ``` The placeholder ID must be replaced with an issued manifest Case ID. This command atomically creates the Group and first child from real results; it does not reserve an empty reference Run. ## Append and complete Use the returned IDs and links. `POST /run-groups/{groupId}/runs` starts another child Trial in that external Group. GET/POST `/run-groups/{groupId}/runs/{runId}/results` reads or appends Case results. Each item contains exactly one output or failure. Optional fields include exact membership/version references, trajectory, latency, usage and cost. Native JSON and whitespace are retained. Accepted Case results cannot be replaced. `POST .../complete` accepts `allow_partial`. Missing Cases require explicit partial completion. Repeated completion returns the recorded outcome; an explicit failed-admission retry does not change accepted content. ## Boundaries Managed and controlled-evaluation Groups reject externally submitted responses. Corrections require a new child Trial. The existing external Group can accept that new child through its registered route; a new Group is also an explicit option. Accepted execution failures remain unavailable response judgments rather than fabricated rubric failures. ## Request sequence Configure `BENCHMARK_URL` with the [quickstart](/docs/public-api/quickstart). Prepare `first-results.json` using the body above, an actual manifest Case ID and its application response. ### Example: Read the manifest and submit first results ```sh curl --fail-with-body \ --header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \ "${BENCHMARK_URL}/case-manifest" curl --fail-with-body \ --header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \ --header 'Content-Type: application/json' \ --header "Idempotency-Key: ${RESULT_OPERATION_KEY}" \ --data-binary @first-results.json \ "${BENCHMARK_URL}/run-groups" ``` ```python import os from pathlib import Path from urllib.parse import quote from urllib.request import HTTPRedirectHandler, Request, build_opener class NoRedirect(HTTPRedirectHandler): def redirect_request(self, req, fp, code, msg, headers, newurl): return None opener = build_opener(NoRedirect()) api_key = os.environ["TEAMMATELY_API_KEY"] url = os.environ["BENCHMARK_URL"] + "/case-manifest" headers = {"Authorization": "Bearer " + api_key} request = Request(url, headers=headers, method="GET") with opener.open(request) as response: print(response.read().decode("utf-8")) url = os.environ["BENCHMARK_URL"] + "/run-groups" headers = {"Authorization": "Bearer " + api_key} headers["Idempotency-Key"] = os.environ["RESULT_OPERATION_KEY"] headers["Content-Type"] = "application/json" body = Path("first-results.json").read_bytes() request = Request(url, data=body, headers=headers, method="POST") with opener.open(request) as response: print(response.read().decode("utf-8")) ``` ```typescript import { readFile } from "node:fs/promises"; const apiKey = process.env.TEAMMATELY_API_KEY!; { const url = new URL(process.env.BENCHMARK_URL! + "/case-manifest"); const response = await fetch(url, { method: "GET", redirect: "error", headers: { Authorization: `Bearer ${apiKey}`, }, }); if (!response.ok) throw new Error(`HTTP ${response.status}`); console.log(await response.text()); } { const url = new URL(process.env.BENCHMARK_URL! + "/run-groups"); const response = await fetch(url, { method: "POST", redirect: "error", headers: { Authorization: `Bearer ${apiKey}`, "Idempotency-Key": process.env.RESULT_OPERATION_KEY!, "Content-Type": "application/json", }, body: new Uint8Array(await readFile("first-results.json")).buffer, }); if (!response.ok) throw new Error(`HTTP ${response.status}`); console.log(await response.text()); } ``` ```go package main import ( "bytes" "context" "fmt" "io" "log" "net/http" "os" ) func main() { if err := run(context.Background()); err != nil { log.Fatal(err) } } func run(ctx context.Context) error { client := &http.Client{ CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } { target := os.Getenv("BENCHMARK_URL") + "/case-manifest" request, err := http.NewRequestWithContext(ctx, "GET", target, nil) if err != nil { return err } request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY")) response, err := client.Do(request) if err != nil { return err } payloadBytes, readErr := io.ReadAll(response.Body) response.Body.Close() if readErr != nil { return readErr } fmt.Println(string(payloadBytes)) if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) } } { target := os.Getenv("BENCHMARK_URL") + "/run-groups" payload, err := os.ReadFile("first-results.json") if err != nil { return err } request, err := http.NewRequestWithContext(ctx, "POST", target, bytes.NewReader(payload)) if err != nil { return err } request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY")) request.Header.Set("Idempotency-Key", os.Getenv("RESULT_OPERATION_KEY")) request.Header.Set("Content-Type", "application/json") response, err := client.Do(request) if err != nil { return err } payloadBytes, readErr := io.ReadAll(response.Body) response.Body.Close() if readErr != nil { return readErr } fmt.Println(string(payloadBytes)) if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) } } return nil } ``` Retain Group and child Run IDs. Append additional batches under the child's `results` route with a key per logical batch. Recover a lost response with its original bytes and key. Read import state before completing. Send `{"allow_partial":false}` to `POST /run-groups/{groupId}/runs/{runId}/complete`; use `true` only for intentionally missing Cases. A failed evaluation-admission retry uses `retry:true` and retains the original partial policy. ## Inspect the result Read `GET /runs/{runId}` for judging progress, then [retrieve results](/docs/public-api/benchmark-results). Missing output, execution failure and an unfavorable rubric judgment are different outcomes. For a full local-file walkthrough, see [CLI external outputs](/docs/cli/external-outputs). ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Retrieve Benchmark Results](/docs/public-api/benchmark-results) - [Result Serialization](/docs/public-api/result-schema) - [API Conventions](/docs/public-api/conventions) {% /related-card-grid %}