# Prepare a Harness for Trialground
Generated: 2026-11-13T08:01:27.301Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: workflow.benchmark-evaluations.prepare-harness
title: Prepare a Harness for Trialground
summary: Prepare an exact runnable candidate before evaluating it against a Benchmark Version.
kind: task
product_area: benchmark_evaluations
status: stable
updated: 2026-11-11
canonical: /docs/benchmark-evaluations/prepare-harness
---
# Prepare a Harness for Trialground
Prepare an exact runnable candidate before evaluating it against a Benchmark Version.
## Prerequisites
A Harness Draft, the intended Case input contract, and access to the required execution settings.
## Steps
1. Inspect the Harness files, entry point, declared Python dependencies, and how it consumes Case input and materials.
2. Use the editor’s validation and debug workflow with supplied inputs to identify packaging or execution problems.
3. Publish the intended Draft as a saved Harness Version. Retain the exact Version identity for the candidate.
4. Open Benchmark Evaluations, choose the intended Benchmark Version, and configure managed execution for that saved candidate.
## Object and state changes
Publishing creates a saved candidate boundary. Debugging supplied inputs and running the benchmark are separate operations.
## Success criteria
The evaluation identifies an exact Harness Version with compatible dependencies and the intended input architecture.
## Common failure modes
Trying to evaluate an editable Draft; assuming a successful debug input establishes benchmark performance.
## Related reference pages
{% related-card-grid title="Continue this workflow" %}
- [Manage Harness Drafts and Versions](/docs/assets/harnesses)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Resolve a problem" %}
- [Permissions troubleshooting](/docs/troubleshooting/permissions)
- [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs)
{% /related-card-grid %}
---
id: assets.harnesses
title: Manage Harness Drafts and Versions
summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
kind: reference
product_area: assets
status: stable
updated: 2026-11-11
canonical: /docs/assets/harnesses
---
# Manage Harness Drafts and Versions
## Definition
A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Lemon Code proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.
A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.
## Fields, states, or lifecycle rules
- A newly created Harness begins with one blank editable Draft.
- Draft source and runtime changes autosave after a short delay. A concurrent edit is reported so you can reload the current Draft and review the conflicting change.
- The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
- Draft validation errors block publication. Warnings remain visible for operator review.
- **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also requests use of that Version for the selected Benchmark; confirm its **In use** state.
- A previously saved Version can be selected with **Use this Version** for the current benchmark.
- Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
- Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
- Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
- Lemon Code proposals appear as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
- Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`.
- Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation results.
## Draft, saved version, and activation
Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.
Use **Publish** or **Use this Version** in the Harness editor to select a saved Version for the current Benchmark. At evaluation launch, choose the saved candidates, machine, and Trials. Each Run retains the Version it executed while your team continues developing the draft.
## Runtime, debug, and output mapping
Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.
Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported candidate responses, Benchmark Evaluations maps uploaded rows to the frozen Cases in the selected Benchmark Version. Inspect the mapped responses and their rubric judgments in Results; Case reference responses remain separate examples of intended behavior.
Lemon Goal Sessions can generate or receive candidate Harness Versions. Those candidates become meaningful in a Session only when their identity and canonical evaluation receipt are observable.
{% example-demo title="Example: exact candidate identity" %}
An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Start a Lemon Goal Session](/docs/improve/start-improvement-session)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
---
id: troubleshooting.permissions
title: Permissions troubleshooting
summary: Separate Project membership from Expert Contribution assignment and readiness when an expert has no tasks available.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/permissions
---
# Permissions troubleshooting
Use this when project role, reviewer assignment, or approval ownership prevents someone from completing the intended work.
## Symptom
A user can access Teammately but cannot open the expected Project, or an expert reports **no tasks available** after opening their Contribution link. These symptoms belong to different owning surfaces.
## Likely causes
- The user lacks the required Project permission through a direct grant, workspace group, or workspace role.
- The Contribution was requested for a different recipient.
- The Contribution exists but its next Task or Checkpoint is not ready.
- The current IP is blocked before Project membership is evaluated.
## Diagnostic checks
- Check direct grants in **Project Settings → Project Members**, the user's workspace-group membership, and the group's permission for the intended Project. Absence from the direct member list alone does not establish lack of access. Inspect workspace status and role in **Admin → Directory → Members** as well.
- Open the Contribution from the sender-side Expert Contributions workspace and verify its recipient and status.
- Check whether the expert has an executable Task or Checkpoint, not merely whether the Contribution exists.
- If **Access Restricted** is visible, resolve the IP allowlist first.
## Fix
- Add the correct individual workspace user under **Project Settings → Project Members** when direct Project membership is missing.
- If access is intended through a workspace group, confirm both the user's membership in that group and its permission for the intended Project.
- Correct the Contribution recipient through the owning Contribution workflow when the request went to the wrong person.
- Resolve lifecycle or readiness problems in Expert Contributions when access succeeds but no Task can be entered.
- Do not infer a permission from a role label or edit governed artifacts merely to make a control appear.
## Prevention
- Separate reviewer access from approval authority in review setup.
- Check assignments before launching a review session.
- Use a small pilot Contribution before assigning a larger specialist cohort.
- Keep project membership changes visible to review owners.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Request an Expert Contribution](/docs/expert-contributions/request-contribution)
- [Expert Contributions](/docs/expert-contributions)
- [Complete an Expert Contribution](/docs/expert-contributions/complete-contribution)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [Permissions](/docs/reference/permissions)
- [Roles and Permissions](/docs/governance/roles-and-permissions)
- [Expert Contributions](/docs/expert-contributions)
{% /related-card-grid %}
---
id: troubleshooting.benchmark-runs
title: Benchmark run troubleshooting
summary: Diagnose a Run that cannot start, has no usable outputs, or produces results that cannot be compared safely.
kind: error
product_area: troubleshooting
status: stable
updated: 2026-11-06
canonical: /docs/troubleshooting/benchmark-runs
---
# Benchmark run troubleshooting
## Symptoms
- The evaluation surface has no Benchmark Version to run.
- Output import or mapping cannot identify a Case or output column.
- A Run is created but remains empty, incomplete, or failed.
- Results appear under the wrong candidate label or Harness Version.
- Two Runs show a score change but do not share a comparable snapshot boundary.
## Likely causes
- No immutable Snapshot exists for the intended evaluation set.
- Imported outputs are mapped to the wrong Case or column.
- The saved Harness Version or candidate metadata does not match the evaluated system.
- The Run is partial, failed, or being compared across different Benchmark Versions.
## Check the boundary before the failure
1. Confirm the URL and page identify the intended Project, Benchmark, and Benchmark Version.
2. Open the Version or Snapshot and verify it contains the expected Cases and evaluators. If it does not, repair coverage and create a new Snapshot; do not edit the historical Run.
3. For managed execution, confirm the saved Harness Version and recorded settings. For imported results, confirm that the external candidate context describes the system that produced the submitted responses.
4. If importing outputs, inspect the mapping preview. Match the Case identifier and candidate-output column deliberately; do not use a reference-output column as candidate behavior.
5. Open Run detail and inspect status, Case count, errors, metadata, and per-Case results before trusting aggregates.
## Fix
- **No runnable version:** finish Case selection and create a Snapshot first.
- **Missing imported outputs:** append only missing Cases to an uncompleted Trial using the exact manifest IDs. Correcting an accepted response or adding Cases after completion requires a new Trial in the external Run or a new Run. Preserve the original results.
- **Wrong Harness or metadata:** create a correctly configured Run. Do not relabel completed results to represent a different system.
- **Partial managed failure:** inspect and preserve the existing Run's recorded outcomes. For another independent execution under unchanged conditions, use **Run again**. **Run evaluation** reopens matching work, including failed work. Publish a new Harness Version when its code changes, or create a new Snapshot and Benchmark Version when Case content or criteria change.
- **Confusing comparison:** compare the Benchmark Version, Harness Version, candidate metadata, and evaluator boundary. Qualify or avoid the comparison when more than the intended variable changed.
## Prevention
Create a Snapshot before evaluation. For managed execution, save the exact Harness Version. For imported results, preserve external candidate context and preview the mapping on representative Cases. Before comparing, confirm that every difference between the two Runs is intentional and visible.
## Recovery check
Open the recovered Run and sample several Case results. Confirm the displayed input, candidate output, reference output where present, applicable Rubrics, and metadata all belong together. A completed status alone does not prove correct mapping.
{% example-demo title="Example: scores drop after output import" %}
A team imports a new candidate file and sees a sudden score collapse. Run detail shows that the column containing reference outputs was mapped as candidate output. The operator creates a new Run, maps the actual candidate column, preserves the mistaken Run as an identifiable failed attempt, and compares only the corrected Run with the prior candidate under the same Benchmark Version.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related workflows" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
- [Inspect Evaluation Results](/docs/benchmark-evaluations/inspect-results)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Related reference" %}
- [Benchmarks and versions](/docs/concepts/benchmarks-and-versions)
- [Benchmark snapshots](/docs/coverage-engineering/benchmark-snapshots)
- [Outputs](/docs/object-model/outputs)
{% /related-card-grid %}