# Work with External Agents
Generated: 2026-11-02T06:36:34.015Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: improve.work-evolve
title: Work with External Agents
summary: Copy a benchmark-scoped task for your coding agent to implement and evaluate in your real application repository.
kind: task
product_area: improve
status: stable
updated: 2026-10-30
canonical: /docs/improve/external-agents
---
# Work with External Agents
External Agents gives your coding agent a concrete improvement task grounded in the selected Benchmark Version. The agent works in your real repository; Teammately supplies the applicable Cases, Policies, Rubrics and recorded results.
## Prerequisites
- A selectable Benchmark Version.
- Access to the application repository and authorization for the intended changes.
- A coding agent connected through the setup supplied with the task.
- An optional reference Run that genuinely represents the application starting point.
## Steps
1. Open **Harness improvement → External Agents**.
2. Select the Benchmark Version. Changing it clears local criterion and reference-Run selections.
3. Focus on the whole Benchmark or choose up to 20 approved and applicable Policy and Rubric Versions.
4. Optionally select one individual reference Run. No Run is silently selected.
5. Choose the allowed improvement direction: supported practices, prompt only, model only, the same model family, or model configuration only.
6. Copy the task, including setup instructions when needed, into your coding-agent conversation.
7. Have the agent inspect the actual repository, retrieve the pinned criteria and Run context, implement the change and evaluate it.
8. Inspect resulting Runs through ordinary Benchmark Evaluations.
The copied task includes the Project Agent Brief when present. Credentials do not belong in that text.
## Execution choices
A Teammately Harness and source upload are optional. Customer-executed responses are the default path for the agent's real application. Managed Harness execution requires a deliberate choice.
The product data MCP supplies read-only context, instructions, results and comparisons. The CLI owns local files and submission operations. The documentation MCP only retrieves these manuals.
Start with [CLI authentication](/docs/cli/authentication) and [product MCP setup](/docs/cli/mcp), then follow [external output submission](/docs/cli/external-outputs) or the optional [managed Harness workflow](/docs/cli/managed-harness).
## Object and state changes
The composer keeps selections locally and copies text. It creates no hosted agent Session, transcript, durable handoff object, automatic code adoption or Harness activation. Later submitted responses or managed execution create their own ordinary Run records.
## Success criteria
- The task preserves the intended Benchmark Version and criteria.
- A selected reference Run is checked against actual code.
- Implementation and comparison use actual pre-change and post-change behavior.
- Accepted responses and candidate identity remain attributable.
## Common failure modes
- Stopping after tool setup or context retrieval.
- Assuming a Harness upload is mandatory.
- Treating copied text as an executed Session.
- Using a reference Run from unrelated code.
- Claiming private agent work before returned artifacts or results exist.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Harness Improvement](/docs/improve)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Project Context](/docs/agent-setup/project-context)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Output mapping troubleshooting](/docs/troubleshooting/output-mapping)
- [Permissions troubleshooting](/docs/troubleshooting/permissions)
{% /related-card-grid %}
---
id: improve.overview
title: Harness Improvement
summary: Improve candidate behavior with Lemon Goal or give your coding agent a scoped External Agents task.
kind: concept
product_area: improve
status: stable
updated: 2026-10-30
canonical: /docs/improve
---
# Harness Improvement
Harness improvement offers two ways to act on benchmark results: **Lemon Goal** coordinates goal-driven work inside Teammately, and **External Agents** prepares a task for your coding agent in your own repository.
## Definition
Lemon Goal starts from a selected saved Harness and measurable criteria. It creates the Goal and its Session together, preserves the original instructions and follows candidate changes through actual evaluations.
External Agents pins a Benchmark Version and optionally selected criteria or a reference Run. It supplies the correctness context while your agent changes the real application. A Teammately Harness or source upload is optional for that path.
## Decision checkpoint
| Need | Choose | Boundary |
| --- | --- | --- |
| Evolve a saved Harness toward numerical targets | Lemon Goal | Recorded Goal, Session, candidates and evaluations |
| Change application code in your own environment | External Agents | Copied task and exact benchmark context |
| Clarify a mistaken standard | Expert Contribution or Correctness Governance | Human judgment and artifact approval |
| Add missing input situations | Coverage Design | Requirements and explicit Case adoption |
## Work with Lemon Goal
Choose the starting Harness, criteria, permitted direction and execution settings. Read actual starting measurements where available; missing measurements stay unmeasured. Starting creates the Goal and begins its Session without a separate user-managed Goal publication process.
Use the conversation to read the agent's conclusions, answer explicit questions and provide authorized steering. Goal, Results, Evolution and Harnesses expose the current purpose, measurements, candidate history and source artifacts.
Completed means the agent concluded its work. Read the reported outcome to determine whether the Goal was achieved. Stopping a Session preserves its recorded work.
## Work with an external agent
Copy the task from External Agents. Your coding agent retrieves the pinned context, inspects your repository and carries out the requested implementation and evaluation. The copied task is not a hosted Teammately agent Session.
Imported responses and managed Harness executions return to ordinary Benchmark Evaluations. Comparison does not automatically adopt code or activate a candidate.
## Keep the responsible object clear
A candidate failure can expose an incorrect Case, an applicability problem or a missing coverage slice. Fixing candidate code is not the answer to every benchmark discrepancy. Preserve the exact candidate and Benchmark Version so the next action can be assessed from the same recorded population.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Start a Lemon Goal Session](/docs/improve/start-improvement-session)
- [Lemon Goals](/docs/improve/goals)
- [Work with External Agents](/docs/improve/external-agents)
- [Candidates and Results](/docs/improve/candidates-and-frontier)
- [Session Conversation and History](/docs/improve/chronology-and-trajectories)
- [Coevolve](/docs/concepts/coevolve)
{% /related-card-grid %}
---
id: benchmark-evaluations.output-mapping
title: Import External Evaluation Outputs
summary: Import native responses or execution failures for exact benchmark Cases and complete the Trial for rubric evaluation.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-10-30
canonical: /docs/benchmark-evaluations/output-mapping
---
# Import External Evaluation Outputs
Import externally produced responses when execution belongs to your application. Teammately associates them with the Cases in the selected Benchmark Version and evaluates them against that Version's criteria.
## Prerequisites
- An immutable Benchmark Version and its issued Case IDs.
- Actual responses or recorded execution failures.
- Permission to import results.
- Explicit intent to complete a partial Trial if some Cases are absent.
## File import
Open **Benchmark Evaluations → Runs** and the import flow. Match the response rows to the exact Case IDs, inspect supplied and expected counts, then confirm **Import and evaluate**.
Files can be CSV, a JSON array or JSONL up to 10 MB. CSV response cells are text. JSON retains native response types and text whitespace. A JSON null response is a submitted value, not an absent response.
## API submission
The in-product API instructions expose the Case manifest and request examples for the selected Version. First submission creates a Run and its first Trial atomically from real results. Merely opening the instructions or choosing a destination does not reserve an empty Run.
Use returned IDs and links for later batches and completion. Reuse the exact idempotency key and request when recovering an uncertain submission. Each result supplies a Case ID and exactly one response or execution failure; recorded trajectories, latency, usage and cost are optional.
## Object and state changes
Accepted Case results are immutable within a Trial. Additional batches can append missing Cases but cannot replace accepted responses. Corrections require a new Trial, either within the same external Run or in a new Run, while retaining the original response.
Completion freezes the submitted population and queues evaluation. Missing Cases require explicit partial completion. A retry of failed evaluation admission does not modify accepted responses.
## Success criteria
- Rows identify Cases in the intended Benchmark Version.
- Supplied, missing and failed-execution counts are understood.
- Completion and evaluation state are confirmed separately.
- Corrected responses are distinguishable from the original import.
## Common failure modes
- Joining by row order or similar text instead of Case identity.
- Reusing an idempotency key for changed content.
- Treating an execution failure as a valid failing response.
- Treating imported responses as an executable Harness.
- Assuming recorded trajectories disclose private reasoning.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Connect Model Outputs](/docs/integrations/connect-model-outputs)
- [Dataset Snapshots](/docs/benchmark-datasets/snapshots)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Output mapping troubleshooting](/docs/troubleshooting/output-mapping)
- [Missing Outputs](/docs/troubleshooting/missing-outputs)
{% /related-card-grid %}
---
id: benchmark-evaluations.compare
title: Evaluation Results
summary: Inspect Case and rubric outcomes and compare exact managed or imported Runs within a Benchmark Version.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-10-30
canonical: /docs/benchmark-evaluations/results
---
# Evaluation Results
Results presents a matrix of selected Runs and the Cases or criteria they evaluated. Use one Run for inspection or several comparable Runs to locate gains, regressions and missing judgments.
## Prerequisites
- At least one Run for the intended Benchmark Version.
- Permission to inspect its Cases, responses and judgments.
- Comparable recorded conditions when interpreting differences.
## Steps
1. Open **Benchmark Evaluations → Results** and confirm the Benchmark Version.
2. Select the visible Runs. Managed and imported Runs can both appear; an imported Run does not need a Harness Version to be inspected.
3. Choose Cases, Policies, Rubrics, Dimension ontology values, Topics, Topic Groups or Scenarios as the row grouping.
4. Read completeness before comparing scores. An empty or incomplete result is not a failed judgment.
5. Open a Case or rubric detail to inspect the response, criterion, observed judgment and rationale.
6. For repeated Trials, inspect the selected aggregation rule and the individual Trial outcomes.
7. Route an actual candidate problem to Harness improvement, a standard problem to Correctness Governance, or an input-coverage problem to Coverage Design.
## Representative and historical Runs
Default candidate results use the Run selected before its outcomes were known. **Used for score** identifies that reference. Other executions remain in Run history and can be inspected without silently replacing the representative result.
Selecting matrix columns changes the analysis view. It does not activate a Harness, execute a candidate or declare a winner.
## Object and state changes
Filtering, row grouping and inspection are read-only. An authorized project manager can separately use **Correct judgment** on an exact rubric result. That operation records a correction and preserves the original AI judgment; refreshed aggregates come from the authoritative result snapshot.
## Success criteria
- Every compared column has an identified Run and Benchmark Version.
- Differences resolve to actual Cases and applicable criteria.
- Missing outcomes stay distinct from negative scores or failed gates.
- Manual corrections remain attributable.
## Common failure modes
- Treating an imported Run as an executable Harness.
- Combining later successful Trials into an earlier reference score.
- Reading a filtered population as the entire benchmark.
- Changing a rubric judgment when the real problem is an incorrect Case or standard.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
- [Evaluation Dashboard](/docs/benchmark-evaluations/dashboard)
- [Rubric Points and Benchmark Scores](/docs/correctness-governance/scoring)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Related troubleshooting pages" %}
- [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly)
- [Overlapping Rubrics](/docs/troubleshooting/overlapping-rubrics)
{% /related-card-grid %}
---
id: agent-setup.project-context
title: Project Context
summary: Maintain the Project Agent Brief that gives Teammately agents stable, project-wide understanding.
kind: reference
product_area: agent_setup
status: stable
updated: 2026-10-30
canonical: /docs/agent-setup/project-context
---
# Project Context
## Definition
Project Context is the Agent Setup surface that edits the **Project Agent Brief**. The brief gives Teammately agents stable project-wide understanding: what the specialist AI is for, which behavior matters, important constraints, terminology, and other context that should carry across coverage, contribution preparation, case construction, and improvement work.
The Project Agent Brief is different from the Project Memo in General settings. The memo is administrative project text and explicitly is not used as prompt or agent context. Put agent-relevant project understanding in Project Context.
## Fields, states, or lifecycle rules
- The brief is scoped to the project and reused across benchmark workspaces.
- Editing the brief changes future agent context; it does not rewrite completed Contributions, Runs, or Improvement Session history.
- The brief provides orientation and constraints, not governed correctness authority. Policies and rubrics remain in Correctness Governance.
- Controlling source material belongs in Reference Materials. Summarize stable project intent in the brief and keep source-backed detail in the indexed material.
- The brief should state product-specific meaning directly. Avoid copying transient benchmark goals, one expert's unconfirmed opinion, or a temporary candidate hypothesis into permanent project context.
## Writing a useful brief
Describe the specialist AI's purpose, users, important domain vocabulary, expected interaction shape, and constraints that affect many workflows. Include explicit boundaries where agents might otherwise make unsafe assumptions. Name controlling authorities without duplicating entire manuals.
Review the brief when the product purpose, domain, input architecture, or correctness boundary changes materially. If only one benchmark needs a special objective, put it in that benchmark's setup or Contribution. If only one Improvement Session needs a constraint, put it in the Goal.
{% example-demo title="Example: project context boundary" %}
The brief states that a procurement assistant supports internal buyers, must distinguish current agreements from expired ones, and should expose uncertainty rather than invent an exception. The current agreements themselves remain indexed Reference Materials. The exact evaluation target for expired-agreement cases belongs to the benchmark and Improvement Session, not the brief.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Product Quickstart](/docs/quickstart)
- [Reference Materials](/docs/agent-setup/reference-materials)
- [Request an Expert Contribution](/docs/expert-contributions/request-contribution)
{% /related-card-grid %}
---
id: troubleshooting.output-mapping
title: Output mapping troubleshooting
summary: Fix external output rows that do not join cleanly to immutable Case IDs in an output-only Run.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/output-mapping
---
# Output mapping troubleshooting
Use this when externally produced responses exist but the mapping preview cannot join them cleanly to the Cases in the selected Benchmark Version.
## Symptom
Rows are unmatched, duplicated, or mapped to the wrong Case ID, or the wrong column was selected as the response. No output-only Run should be trusted until the mapping preview is correct.
## Likely causes
- External rows use row order or display text instead of immutable Case IDs.
- The file contains Case IDs from another Benchmark Version or Dataset export.
- The selected response column or identifier column is wrong.
- Several candidate output sets were combined without distinct metadata.
## Diagnostic checks
- Compare several affected case IDs with the output source IDs.
- Check whether the output set has candidate metadata such as model, prompt, retrieval version, or run identity.
- Confirm the source file was produced from the same immutable Case population.
- Inspect duplicate, blank, and unknown identifier counts in the mapping preview.
## Fix
- Correct the external file and map using immutable Case IDs rather than row position.
- Select the intended response column and keep one candidate output set per output-only Run.
- Create a new output-only Run from the corrected mapping. Existing Runs remain supporting information under their original mapping.
- Repair missing Case context through the Case workflow, not during output mapping; create a new version boundary when Case content changes.
## Prevention
- Preserve source IDs through import and output generation.
- Sample mapped outputs before expert contribution work and before Evaluation Runs.
- Keep candidate identity in run metadata.
- Treat each imported output-only Run as its own supporting information record; it is not a saved Harness Version.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Import and prepare cases](/docs/operating-manual/import-and-prepare-cases)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Outputs](/docs/object-model/outputs)
- [Represent conversations in Cases](/docs/object-model/represent-conversations-in-cases)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
{% /related-card-grid %}
---
id: troubleshooting.permissions
title: Permissions troubleshooting
summary: Separate Project membership from Expert Contribution assignment and readiness when an expert has no tasks available.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-10-30
canonical: /docs/troubleshooting/permissions
---
# Permissions troubleshooting
Use this when project role, reviewer assignment, or approval ownership prevents someone from completing the intended work.
## Symptom
A user can access Teammately but cannot open the expected Project, or an expert reports **no tasks available** after opening their Contribution link. These symptoms belong to different owning surfaces.
## Likely causes
- The user or group is not listed under **Project Settings → Project Members**.
- The Contribution was requested for a different recipient.
- The Contribution exists but its next Task or Checkpoint is not ready.
- The current IP is blocked before Project membership is evaluated.
## Diagnostic checks
- Confirm that the user or group appears in **Project Members** for the intended Project.
- Open the Contribution from the sender-side Expert Contributions workspace and verify its recipient and status.
- Check whether the expert has an executable Task or Checkpoint, not merely whether the Contribution exists.
- If **Access Restricted** is visible, resolve the IP allowlist first.
## Fix
- Add the correct user or group under **Project Settings → Project Members** when Project access is missing.
- Correct the Contribution recipient through the owning Contribution workflow when the request went to the wrong person.
- Resolve lifecycle or readiness problems in Expert Contributions when access succeeds but no Task can be entered.
- Do not infer a permission from a role label or edit governed artifacts merely to make a control appear.
## Prevention
- Separate reviewer access from approval authority in review setup.
- Check assignments before launching a review session.
- Use a small pilot Contribution before assigning a larger specialist cohort.
- Keep project membership changes visible to review owners.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Request an Expert Contribution](/docs/expert-contributions/request-contribution)
- [Expert Contributions](/docs/expert-contributions)
- [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Permissions](/docs/reference/permissions)
- [Roles and Permissions](/docs/governance/roles-and-permissions)
- [Expert Contributions](/docs/expert-contributions)
{% /related-card-grid %}