Submit External Evaluation Outputs
Definition
Use the organization/project/Benchmark/Version prefix and retrieve GET /case-manifest for exact frozen Cases. Product Runs and Trials retain the API's existing run-groups and child runs wire identities.
Create with actual results
The placeholder ID must be replaced with an issued manifest Case ID. This command atomically creates the Group and first child from real results; it does not reserve an empty reference Run.
POST /run-groups
Idempotency-Key: <logical-command-key>
{
"name": "Application comparison",
"results": [
{
"case_id": "<issued-case-id>",
"output": "<response produced by the application>"
}
]
}
Append and complete
Use the returned IDs and links. POST /run-groups/{groupId}/runs starts another child Trial in that external Group. GET/POST /run-groups/{groupId}/runs/{runId}/results reads or appends Case results.
Each item contains exactly one output or failure. Optional fields include exact membership/version references, trajectory, latency, usage and cost. Native JSON and whitespace are retained. Accepted Case results cannot be replaced.
POST .../complete accepts allow_partial. Missing Cases require explicit partial completion. Repeated completion returns the recorded outcome; an explicit failed-admission retry does not change accepted content.
Boundaries
Managed and controlled-evaluation Groups reject externally submitted responses. Corrections require a new child Trial. The existing external Group can accept that new child through its registered route; a new Group is also an explicit option. Accepted execution failures remain unavailable response judgments rather than fabricated rubric failures.
Request sequence
Configure BENCHMARK_URL with the quickstart. Prepare first-results.json using the body above, an actual manifest Case ID and its application response.
Retain Group and child Run IDs. Append additional batches under the child's results route with a key per logical batch. Recover a lost response with its original bytes and key.
Read import state before completing. Send {"allow_partial":false} to POST /run-groups/{groupId}/runs/{runId}/complete; use true only for intentionally missing Cases. A failed evaluation-admission retry uses retry:true and retains the original partial policy.
curl --fail-with-body \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
"${BENCHMARK_URL}/case-manifest"
curl --fail-with-body \
--header "Authorization: Bearer ${TEAMMATELY_API_KEY}" \
--header 'Content-Type: application/json' \
--header "Idempotency-Key: ${RESULT_OPERATION_KEY}" \
--data-binary @first-results.json \
"${BENCHMARK_URL}/run-groups"
Inspect the result
Read GET /runs/{runId} for judging progress, then retrieve results. Missing output, execution failure and an unfavorable rubric judgment are different outcomes. For a full local-file walkthrough, see CLI external outputs.