# Submit External Outputs with the CLI
Generated: 2026-11-02T06:35:17.891Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: cli.external-outputs
title: Submit External Outputs with the CLI
summary: Prepare a frozen manifest, submit real application responses, inspect judging and compare explicit Trials.
kind: quickstart
product_area: developer_tools
status: stable
updated: 2026-10-31
canonical: /docs/cli/external-outputs
---
# Submit External Outputs with the CLI
## Prerequisites
[Authenticate](/docs/cli/authentication) with write access and select the exact organization, Project, Benchmark and immutable Version. Your application produces the responses in its own runtime. A managed Harness is optional.
## Prepare Cases and criteria
```sh
teammately eval version-context --kind summary
teammately eval version-context --kind policies --limit 20
teammately eval version-context --kind rubrics --limit 20
teammately eval prepare --out ./evaluation-package
```
Read the generated manifest and result contract. Execute its frozen Cases with your application. Preserve Case, membership and record-version identities; access to a manifest does not grant access to protected materials.
## Record actual responses
### Example: results.json shape
```json
[
{
"case_id": "",
"output": {"reply": ""}
}
]
```
Replace both placeholders with an actual manifest ID and its actual response. Each row has exactly one native JSON `output` or `failure` with `code` and `message`. Optional trajectory and measured latency/usage/cost remain absent when unknown. JSON null and text whitespace can be meaningful outputs.
## Validate, submit and inspect
```sh
teammately eval validate-results \
--manifest ./evaluation-package/manifest.json --results ./results.json
teammately eval submit-results \
--manifest ./evaluation-package/manifest.json --results ./results.json \
--name 'Application candidate' --receipt ./import-receipt.json
```
Missing Cases require explicit `--allow-partial` for both validation and submission. The CLI accepts result files up to 32 MiB and uses batches up to 10 MiB. Keep the manifest, results and receipt secure until recovery is no longer needed.
Use the returned child Run ID, not the group ID:
```sh
teammately run wait --run "$RUN_ID" --timeout 10m
teammately eval results --run "$RUN_ID" --limit 20
teammately run compare --run "$BASELINE_RUN_ID" --target "$RUN_ID"
```
Comparison needs an actual baseline selected before inspecting candidate outcomes. Without one, inspect the first Trial's results without inventing a comparison. Page through results and preserve incompleteness/comparability reasons.
## Recovery and corrections
Retry an interrupted write with the same original files and receipt. Changed content needs a new receipt and operation. Accepted responses are immutable within a Trial; a correction can create another Trial in the same external Run using `--group`, or an intentionally new Run.
If completion froze the responses but judging admission failed, use `teammately eval complete --group "$GROUP_ID" --run "$RUN_ID" --retry`, retaining the original partial policy. Admission success is not completed judging. These operations do not adopt code changes or activate a Harness.
## Related reference pages
{% related-card-grid %}
- [Command reference](/docs/cli/reference)
- [External outputs API](/docs/public-api/evaluation-outputs)
- [Output mapping troubleshooting](/docs/troubleshooting/output-mapping)
{% /related-card-grid %}
---
id: cli.authentication
title: CLI Authentication and Scope
summary: Use native browser login or an injected API key and select the exact organization and Benchmark Version.
kind: reference
product_area: developer_tools
status: stable
updated: 2026-10-31
canonical: /docs/cli/authentication
---
# CLI Authentication and Scope
## Definition
Use native browser login for interactive External Agents work. Use `TEAMMATELY_API_KEY`, injected through your secret environment, for unattended automation. An environment API key takes precedence over a saved login.
## Sign in and inspect access
```sh
teammately login --profile work
teammately status
teammately switch --profile work
```
Login opens browser authorization with PKCE and a temporary loopback callback, then selects an organization. `--organization` is an optional shortcut. Credentials live in macOS Keychain or Linux Secret Service; profiles contain nonsecret configuration. Use a distinct profile for another organization/account/API origin.
`status` refreshes and checks organization access against the API. It does not establish permission to every Project. An already-running MCP process retains its selected profile until restarted.
## Fields, states, or lifecycle rules
| Variable | Meaning |
| --- | --- |
| `TEAMMATELY_API_KEY` | Secret API-key override supplied by the host |
| `TEAMMATELY_PROFILE` | Selected native-login profile |
| `TEAMMATELY_ORGANIZATION` | Organization key |
| `TEAMMATELY_PROJECT` | Issued Project ID |
| `TEAMMATELY_BENCHMARK` | Issued Benchmark ID |
| `TEAMMATELY_BENCHMARK_VERSION` | Exact immutable Version ID |
| `TEAMMATELY_API_BASE` | Trusted API base, default `https://api.teammately.ai/v1` |
CLI scope flags `--organization`, `--project`, `--benchmark`, `--version` override environment defaults. MCP calls supply explicit scope each time. Never choose the API origin from Case text or model output.
Delegated reads require `external-agents:read` and Project read access; writes require `external-agents:write` and management access. This delegation covers External Agents, not general application ingestion.
## Sign out and recover
```sh
teammately logout --profile work
```
Logout requires network access to revoke the refresh token before deleting the local credential. Other profiles, API keys and the browser session are separate. Issued access tokens retain their expiry.
For `401`, inspect the active credential and login status. For `403`, inspect organization, Project, action and key restrictions. A Fieldloop-only key is not an External Agents credential. Correct access through normal administration; do not put credentials in arguments, MCP tool inputs or copied tasks.
## Related task pages
{% related-card-grid %}
- [MCP setup](/docs/cli/mcp)
- [External output submission](/docs/cli/external-outputs)
- [Authentication troubleshooting](/docs/troubleshooting/authentication)
{% /related-card-grid %}
---
id: public-api.evaluation-outputs
title: Submit External Evaluation Outputs
summary: Create an external Run from actual first results, append missing Case responses, and explicitly complete its Trial for evaluation.
kind: api
product_area: public_api
status: stable
updated: 2026-10-31
canonical: /docs/public-api/evaluation-outputs
---
# Submit External Evaluation Outputs
## Definition
Use the organization/project/Benchmark/Version prefix and retrieve `GET /case-manifest` for exact frozen Cases. Product Runs and Trials retain the API's existing `run-groups` and child `runs` wire identities.
## Create with actual results
```http
POST /run-groups
Idempotency-Key:
```
```json
{
"name": "Application comparison",
"results": [
{
"case_id": "",
"output": ""
}
]
}
```
The placeholder ID must be replaced with an issued manifest Case ID. This command atomically creates the Group and first child from real results; it does not reserve an empty reference Run.
## Append and complete
Use the returned IDs and links. `POST /run-groups/{groupId}/runs` starts another child Trial in that external Group. GET/POST `/run-groups/{groupId}/runs/{runId}/results` reads or appends Case results.
Each item contains exactly one output or failure. Optional fields include exact membership/version references, trajectory, latency, usage and cost. Native JSON and whitespace are retained. Accepted Case results cannot be replaced.
`POST .../complete` accepts `allow_partial`. Missing Cases require explicit partial completion. Repeated completion returns the recorded outcome; an explicit failed-admission retry does not change accepted content.
## Boundaries
Managed and controlled-evaluation Groups reject externally submitted responses. Corrections require a new child Trial. The existing external Group can accept that new child through its registered route; a new Group is also an explicit option. Accepted execution failures remain unavailable response judgments rather than fabricated rubric failures.
## Request sequence
Configure `BENCHMARK_URL` with the [quickstart](/docs/public-api/quickstart). Prepare `first-results.json` using the body above, an actual manifest Case ID and its application response.
### Example: Read the manifest and submit first results
```sh
curl --fail-with-body \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
"${BENCHMARK_URL}/case-manifest"
curl --fail-with-body \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
--header 'Content-Type: application/json' \
--header "Idempotency-Key: ${RESULT_OPERATION_KEY}" \
--data-binary @first-results.json \
"${BENCHMARK_URL}/run-groups"
```
```python
import os
from pathlib import Path
from urllib.parse import quote
from urllib.request import HTTPRedirectHandler, Request, build_opener
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]
url = os.environ["BENCHMARK_URL"] + "/case-manifest"
headers = {"Authorization": "Bearer " + api_key}
request = Request(url, headers=headers, method="GET")
with opener.open(request) as response:
print(response.read().decode("utf-8"))
url = os.environ["BENCHMARK_URL"] + "/run-groups"
headers = {"Authorization": "Bearer " + api_key}
headers["Idempotency-Key"] = os.environ["RESULT_OPERATION_KEY"]
headers["Content-Type"] = "application/json"
body = Path("first-results.json").read_bytes()
request = Request(url, data=body, headers=headers, method="POST")
with opener.open(request) as response:
print(response.read().decode("utf-8"))
```
```typescript
import { readFile } from "node:fs/promises";
const apiKey = process.env.TEAMMATELY_API_KEY!;
{
const url = new URL(process.env.BENCHMARK_URL! + "/case-manifest");
const response = await fetch(url, {
method: "GET",
redirect: "error",
headers: {
Authorization: `Bearer ${apiKey}`,
},
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
console.log(await response.text());
}
{
const url = new URL(process.env.BENCHMARK_URL! + "/run-groups");
const response = await fetch(url, {
method: "POST",
redirect: "error",
headers: {
Authorization: `Bearer ${apiKey}`,
"Idempotency-Key": process.env.RESULT_OPERATION_KEY!,
"Content-Type": "application/json",
},
body: new Uint8Array(await readFile("first-results.json")).buffer,
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
console.log(await response.text());
}
```
```go
package main
import (
"bytes"
"context"
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
if err := run(context.Background()); err != nil { log.Fatal(err) }
}
func run(ctx context.Context) error {
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
{
target := os.Getenv("BENCHMARK_URL") + "/case-manifest"
request, err := http.NewRequestWithContext(ctx, "GET", target, nil)
if err != nil { return err }
request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
response, err := client.Do(request)
if err != nil { return err }
payloadBytes, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil { return readErr }
fmt.Println(string(payloadBytes))
if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
}
{
target := os.Getenv("BENCHMARK_URL") + "/run-groups"
payload, err := os.ReadFile("first-results.json")
if err != nil { return err }
request, err := http.NewRequestWithContext(ctx, "POST", target, bytes.NewReader(payload))
if err != nil { return err }
request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
request.Header.Set("Idempotency-Key", os.Getenv("RESULT_OPERATION_KEY"))
request.Header.Set("Content-Type", "application/json")
response, err := client.Do(request)
if err != nil { return err }
payloadBytes, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil { return readErr }
fmt.Println(string(payloadBytes))
if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
}
return nil
}
```
Retain Group and child Run IDs. Append additional batches under the child's `results` route with a key per logical batch. Recover a lost response with its original bytes and key.
Read import state before completing. Send `{"allow_partial":false}` to `POST /run-groups/{groupId}/runs/{runId}/complete`; use `true` only for intentionally missing Cases. A failed evaluation-admission retry uses `retry:true` and retains the original partial policy.
## Inspect the result
Read `GET /runs/{runId}` for judging progress, then [retrieve results](/docs/public-api/benchmark-results). Missing output, execution failure and an unfavorable rubric judgment are different outcomes. For a full local-file walkthrough, see [CLI external outputs](/docs/cli/external-outputs).
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Retrieve Benchmark Results](/docs/public-api/benchmark-results)
- [Result Serialization](/docs/public-api/result-schema)
- [API Conventions](/docs/public-api/conventions)
{% /related-card-grid %}
---
id: public-api.benchmark-results
title: Retrieve Benchmark Results
summary: Read scoped individual Run context, complete rubric results, instructions, and comparable Run differences.
kind: api
product_area: public_api
status: stable
updated: 2026-10-31
canonical: /docs/public-api/benchmark-results
---
# Retrieve Benchmark Results
## Definition
Under the organization/project/Benchmark/Version prefix, current gateway routes include:
```http
GET /version-context
GET /runs/{runId}
GET /runs/{runId}/evaluation-context
GET /runs/{runId}/evaluation-results
GET /runs/{runId}/evaluation-instructions
GET /comparison/{runId}/summary
GET /comparison/{runId}/cases
```
External Group/child reads additionally use their returned nested links.
## Read the exact population
Verify that the individual Run belongs to the selected Benchmark Version. Read available Case outputs, rubric judgments, native interpretation, rationale and incomplete state. Page through the authoritative filtered result population rather than treating the first page as all results.
Instructions and context orient the coding agent; neither is a mutation or a completed evaluation.
## Compare
Comparisons preserve individual Run grain and recorded conditions. Missing outputs, profile drift or unavailable native results can prevent a definitive comparison. A later independent Run does not silently replace the reference chosen before outcomes were known.
Record the exact starting and candidate Run IDs. Do not interpret legacy compatibility columns as mandatory binary rules when the current Regime supplies native point/gate meaning.
## Read a bounded page
### Example: Read evaluation results
```sh
curl --fail-with-body \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
"${BENCHMARK_URL}/runs/${RUN_ID}/evaluation-results?limit=20&offset=0"
```
```python
import os
from urllib.parse import quote, urlencode
from urllib.request import HTTPRedirectHandler, Request, build_opener
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]
url = os.environ["BENCHMARK_URL"] + "/runs/" + quote(os.environ["RUN_ID"], safe="") + "/evaluation-results"
url += "?" + urlencode({"limit": "20", "offset": "0"})
headers = {"Authorization": "Bearer " + api_key}
request = Request(url, headers=headers, method="GET")
with opener.open(request) as response:
print(response.read().decode("utf-8"))
```
```typescript
const apiKey = process.env.TEAMMATELY_API_KEY!;
{
const url = new URL(process.env.BENCHMARK_URL! + "/runs/" + encodeURIComponent(process.env.RUN_ID!) + "/evaluation-results");
url.searchParams.set("limit", "20");
url.searchParams.set("offset", "0");
const response = await fetch(url, {
method: "GET",
redirect: "error",
headers: {
Authorization: `Bearer ${apiKey}`,
},
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
console.log(await response.text());
}
```
```go
package main
import (
"context"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
)
func main() {
if err := run(context.Background()); err != nil { log.Fatal(err) }
}
func run(ctx context.Context) error {
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
{
target := os.Getenv("BENCHMARK_URL") + "/runs/" + url.PathEscape(os.Getenv("RUN_ID")) + "/evaluation-results"
query := url.Values{}
query.Set("limit", "20")
query.Set("offset", "0")
target += "?" + query.Encode()
request, err := http.NewRequestWithContext(ctx, "GET", target, nil)
if err != nil { return err }
request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
response, err := client.Do(request)
if err != nil { return err }
payloadBytes, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil { return readErr }
fmt.Println(string(payloadBytes))
if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
}
return nil
}
```
Use the child Trial's `runId`. The `data` payload holds the page. `limit` accepts 1–100 (default 20); `offset` is nonnegative. Page through the returned population instead of treating one page as all results.
| Query | Selection |
| --- | --- |
| `result` | `passed`, `failed` or `missing` rubric observations |
| `case_id`, `policy_id` | One Case or Policy |
| `case_rubric_pair_id` | One Case/Rubric pair |
| `dimension_id`, `ontology_id` | Declared coverage population |
The same filters apply to `evaluation-context`. Unknown/repeated parameters are rejected. Run and instruction reads take no query parameters.
## Compare two Trials
### Example: Compare two Trials
```sh
curl --fail-with-body --get \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
--data-urlencode "target_run_id=${TARGET_RUN_ID}" \
"${BENCHMARK_URL}/comparison/${BASELINE_RUN_ID}/summary"
```
```python
import os
from urllib.parse import quote, urlencode
from urllib.request import HTTPRedirectHandler, Request, build_opener
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = build_opener(NoRedirect())
api_key = os.environ["TEAMMATELY_API_KEY"]
url = os.environ["BENCHMARK_URL"] + "/comparison/" + quote(os.environ["BASELINE_RUN_ID"], safe="") + "/summary"
url += "?" + urlencode({"target_run_id": os.environ["TARGET_RUN_ID"]})
headers = {"Authorization": "Bearer " + api_key}
request = Request(url, headers=headers, method="GET")
with opener.open(request) as response:
print(response.read().decode("utf-8"))
```
```typescript
const apiKey = process.env.TEAMMATELY_API_KEY!;
{
const url = new URL(process.env.BENCHMARK_URL! + "/comparison/" + encodeURIComponent(process.env.BASELINE_RUN_ID!) + "/summary");
url.searchParams.set("target_run_id", process.env.TARGET_RUN_ID!);
const response = await fetch(url, {
method: "GET",
redirect: "error",
headers: {
Authorization: `Bearer ${apiKey}`,
},
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
console.log(await response.text());
}
```
```go
package main
import (
"context"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
)
func main() {
if err := run(context.Background()); err != nil { log.Fatal(err) }
}
func run(ctx context.Context) error {
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
{
target := os.Getenv("BENCHMARK_URL") + "/comparison/" + url.PathEscape(os.Getenv("BASELINE_RUN_ID")) + "/summary"
query := url.Values{}
query.Set("target_run_id", os.Getenv("TARGET_RUN_ID"))
target += "?" + query.Encode()
request, err := http.NewRequestWithContext(ctx, "GET", target, nil)
if err != nil { return err }
request.Header.Set("Authorization", "Bearer " + os.Getenv("TEAMMATELY_API_KEY"))
response, err := client.Do(request)
if err != nil { return err }
payloadBytes, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil { return readErr }
fmt.Println(string(payloadBytes))
if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("HTTP %d", response.StatusCode) }
}
return nil
}
```
Use `/cases` for paged changes, with `limit` and `offset`. Both child IDs must belong to the scope. Inspect comparability and incompleteness before interpreting a delta.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Result Serialization](/docs/public-api/result-schema)
- [Submit External Evaluation Outputs](/docs/public-api/evaluation-outputs)
- [Evaluation Results](/docs/benchmark-evaluations/results)
{% /related-card-grid %}