# Lemon Code Generated: 2026-11-06T00:55:24.349Z Source build: local Canonical docs: https://teammately.ai/docs --- id: concepts.lemon-code title: Lemon Code summary: Develop a Harness through a conversation connected to its draft, relevant context, and proposed file changes. kind: concept product_area: introduction status: stable updated: 2026-11-06 canonical: /docs/concepts/lemon-code --- # Lemon Code Work with Lemon Code on the code, context, and instructions around your model. The conversation stays connected to the Harness draft, so you can investigate behavior and develop changes where the implementation lives. ## Definition Open a Harness under **Assets → Harnesses**. Describe the behavior you want to understand or change and bring the relevant context into the conversation. Lemon Code can examine the implementation and propose file changes. Inspect the proposed diff, refine the approach through discussion, and decide which changes to apply to the draft. Publishing preserves an exact saved Harness Version for subsequent execution. ## Decision checkpoint | Your question | Work with | Useful result | | --- | --- | --- | | Why does this Harness behave this way? | Lemon Code and the current source | An explanation connected to the implementation | | How should this intervention be implemented? | Lemon Code and selected context | A proposed change for review | | Does the saved candidate behave better? | Trialground | Recorded responses and rubric results | | Which competing approach should we investigate? | Lemon Goal | Candidates tested against a benchmark objective | ## Develop a focused change Explain the observed problem and the desired outcome. Attach the relevant policy, reference material, or benchmark context when it helps establish what the change should achieve. Read the proposal as an implementation decision. Check the affected files, the intended behavior, and the constraints your team needs to preserve. Continue the conversation when the approach needs refinement. ## Take the candidate into evaluation Apply the changes you choose and publish the draft as a saved Version. Use a debug run to try supplied inputs, then run the candidate against the intended Benchmark Version in Trialground. Inspect gains and regressions against the same criteria that motivated the edit. Your engineers decide which ideas to integrate into the production system. {% example-demo title="Resolve conflicting-document answers" %} A benchmark exposes unsupported blends of current and superseded documents. In the Harness editor, ask Lemon Code to examine source selection and propose an intervention. Review the file diff, publish the candidate, and compare its grounding and uncertainty results in Trialground. {% /example-demo %} ## Related workflows {% related-card-grid title="Develop and assess a Harness" %} - [Harnesses](/docs/assets/harnesses) - [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation) - [Start a Lemon Goal Session](/docs/improve/start-improvement-session) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Supply useful context" %} - [Working with Lemon](/docs/concepts/lemon) - [Reference Materials](/docs/agent-setup/reference-materials) - [Rubrics and Policies](/docs/correctness-governance/policies-and-rubrics) {% /related-card-grid %} --- id: assets.harnesses title: Harnesses summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations. kind: reference product_area: assets status: stable updated: 2026-11-06 canonical: /docs/assets/harnesses --- # Harnesses ## Definition A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Lemon Code proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation. A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry. ## Fields, states, or lifecycle rules - A newly created Harness begins with one blank editable Draft. - Draft source and runtime changes autosave after a short delay. A concurrent edit is reported so you can reload the current Draft and review the conflicting change. - The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts. - Draft validation errors block publication. Warnings remain visible for operator review. - **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also requests use of that Version for the selected Benchmark; confirm its **In use** state. - A previously saved Version can be selected with **Use this Version** for the current benchmark. - Existing Runs keep their referenced Harness version even after later Draft or Saved changes. - Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets. - Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata. - Lemon Code proposals appear as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft. - Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`. - Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation results. ## Draft, saved version, and activation Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began. Use **Publish** or **Use this Version** in the Harness editor to select a saved Version for the current Benchmark. At evaluation launch, choose the saved candidates, machine, and Trials. Each Run retains the Version it executed while your team continues developing the draft. ## Runtime, debug, and output mapping Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run. Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported candidate responses, Benchmark Evaluations maps uploaded rows to the frozen Cases in the selected Benchmark Version. Inspect the mapped responses and their rubric judgments in Results; Case reference responses remain separate examples of intended behavior. Lemon Goal Sessions can generate or receive candidate Harness Versions. Those candidates become meaningful in a Session only when their identity and canonical evaluation receipt are observable. {% example-demo title="Example: exact candidate identity" %} An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work. {% /example-demo %} ## Related task pages {% related-card-grid title="Related task pages" %} - [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation) - [Start a Lemon Goal Session](/docs/improve/start-improvement-session) - [Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} --- id: concepts.lemon title: Working with Lemon summary: Work with Lemon on expert elicitation, coverage, Case construction, Harness development, and benchmark-driven improvement. kind: concept product_area: introduction status: stable updated: 2026-11-06 canonical: /docs/concepts/lemon --- # Working with Lemon Lemon is your AI agent for alignment. Bring it an expert question, a coverage need, or a Harness to develop. Lemon undertakes the preparation, inquiry, construction, and investigation around that objective, returning work your team can inspect and direct. ## Definition **Lemon** helps develop coverage Dimensions, Topics, Scenarios, and project wording. **Lemon Elicit** prepares and adapts expert contributions, follows the reasoning behind a judgment, and develops findings into proposed policies, rubrics, and coverage. **Lemon Weave** constructs Cases, controlled alternatives, supporting materials, and targeted training examples. **Lemon Code** supports interactive Harness development through a conversation connected to the draft. **Lemon Goal** investigates benchmark failures, develops competing Harness candidates, and tests them through Trialground while keeping protected behavior and permitted changes in view. Task titles and project or benchmark context identify the work you are viewing. The AI Work surface brings activity, findings, and requests for attention into view. ## Decision checkpoint | Your task | Lemon capability | Where to work | What to inspect | | --- | --- | --- | --- | | Develop coverage structure | Lemon assistance | Coverage Facets and Coverage Design | Proposed facets and representation | | Understand expert judgment | Lemon Elicit | Expert Contributions | Expert responses and proposed interpretations | | Construct missing situations | Lemon Weave | Coverage Design → Canvas | Cases and supporting material | | Develop a Harness interactively | Lemon Code | Assets → Harnesses | Attached context, proposed changes, and the draft | | Pursue a measured objective | Lemon Goal | Harness improvement → Lemon Goal | Candidates, results, and Session conversation | | Prepare targeted teaching material | Lemon Weave | Weights training | Dataset records, splits, coverage, and training results | Capability names describe Lemon's work. Destinations such as Expert Contributions, Harnesses, and Coverage Design identify the workspace where you direct and inspect it. Trialground supplies managed execution and rubric evaluation for the resulting experiments. ## Give the work a useful starting point Project Context gives Lemon the Project Agent Brief. Reference Materials supplies indexed project knowledge. Relevant Cases, policies, rubrics, and results make an objective concrete. For expert work, state what the team needs to learn. For coverage, identify which situation is missing or poorly represented. For Harness development, explain the intended behavior and the constraints the implementation should respect. ## Review the result where it belongs Inspect a proposed Dimension before adopting it. Review a Harness change before applying it to the draft. Check generated Cases against their intended coverage. Experts inspect the interpretation of their reasoning before it becomes a shared standard. Evaluation results stay connected to their criteria and configuration. In Weights training, follow material through dataset preparation, download or connected training, and separate assessment of the trained candidate. {% example-demo title="From a coverage question to a Harness edit" %} Ask Lemon to propose Dimensions for conflicting-document questions. Review the proposal, then find existing Cases or construct missing situations with Lemon Weave. Lemon Elicit helps a specialist clarify which source controls. In the Harness editor, ask Lemon Code to develop a source-selection change and inspect its diff before applying it. {% /example-demo %} ## Related workflows {% related-card-grid title="Work with Lemon" %} - [Lemon Elicit](/docs/concepts/correctness-elicitation) - [Lemon Weave](/docs/concepts/weave) - [Lemon Code](/docs/concepts/lemon-code) - [Lemon Goal](/docs/concepts/lemon-goal) - [Weights Training](/docs/weights-training) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Supply project context" %} - [Project Context](/docs/agent-setup/project-context) - [Reference Materials](/docs/agent-setup/reference-materials) - [Lemon Instructions](/docs/project-settings/lemon-instructions) {% /related-card-grid %} --- id: improve.start-session title: Start a Lemon Goal Session summary: Choose a saved Harness, numerical criteria, permitted changes, and execution settings, then start goal-driven work. kind: task product_area: improve status: stable updated: 2026-10-30 canonical: /docs/improve/start-improvement-session --- # Start a Lemon Goal Session Start a Session when you want Teammately to improve a saved Harness toward an explicit measured outcome. ## Prerequisites - A selected Benchmark Version and saved starting Harness. - Permission to start work in that scope. - Measurable criteria and a clear boundary for permitted changes. - An available execution machine and model choice from the supplied catalog. ## Before and after | Before | After | | --- | --- | | A saved candidate and intended change | An accepted Goal with exact criteria | | Available starting measurements | Recorded starting context, with missing values kept unmeasured | | Free-form instructions | Original instructions attached to the new Session | | Candidate work has not started | The Goal and Session are created together | ## Steps 1. Open **Harness improvement → Lemon Goal**. 2. Choose the saved Harness Version to improve. 3. Add numerical criteria and check their supported conditions, units and target values. 4. Select the permitted improvement direction and write any additional instructions. 5. Review available starting measurements. Missing initial results do not require inventing a baseline. 6. Choose the machine, execution count, candidate settings and available model option. 7. Review the selected scope and start the Session. 8. Follow the conversation. Answer an explicit question when one appears, and inspect Goal, Results, Evolution and Harnesses as work progresses. The agent chooses useful actions within the accepted scope. Selecting a direction does not prescribe a fixed sequence of semantic steps. ## Object and state changes Starting records the Goal and Session atomically. Subsequent work can produce candidate Harness Versions, evaluations, findings and usage records. Accepted inputs remain attached to that work rather than being rewritten by later project settings. ## Success criteria - Criteria refer to the intended objects and preserve their actual units. - The saved starting Version and original instructions are identifiable. - Candidate conclusions refer to actual recorded measurements. - Completion is interpreted from the reported outcome, not status alone. ## Common failure modes - Selecting a different saved Version than intended. - Mistaking unmeasured values for zero. - Using a rounded display as the exact target formula. - Treating Session completion as proof that every criterion was achieved. - Using candidate work to hide a mistaken Case or Rubric. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Lemon Goals](/docs/improve/goals) - [Candidates and Results](/docs/improve/candidates-and-frontier) - [Session Conversation and History](/docs/improve/chronology-and-trajectories) - [Evaluation Results](/docs/benchmark-evaluations/results) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs) - [Benchmark Results Changed Unexpectedly](/docs/troubleshooting/benchmark-results-changed-unexpectedly) {% /related-card-grid %} --- id: benchmark-evaluations.run title: Run a Benchmark Evaluation summary: Choose saved Harness Versions, an execution machine, and Trials, then launch or reuse a comparable Run. kind: task product_area: benchmark_evaluations status: stable updated: 2026-10-30 canonical: /docs/benchmark-evaluations/run-evaluation --- # Run a Benchmark Evaluation Use a managed evaluation to execute a saved Harness Version against every Case in the selected Benchmark Version. ## Prerequisites - A Benchmark Version with the intended Cases and eligible evaluation rules. - A saved Harness Version with its runtime and required secret grants. - Permission to run evaluations and an available execution machine. ## Steps 1. Open the Benchmark's **Benchmark Evaluations → Runs**. 2. Choose **Run evaluation** and select the saved Harness Versions. Confirm each Version rather than relying on the Harness name. 3. Select the execution machine. Its CPU and memory allocation must meet the Harness requirements. 4. Keep one Trial for a single execution, or select **Repeat Trials** and enter the total Trials per Run, including the first. 5. Check the displayed Case executions and confirm the launch. 6. Follow output preparation and rubric evaluation separately. Responses can finish before their judgments. 7. Open Results after checking the planned population and completeness. The service accepts 1–50 Trials per selected candidate. The launch form checks a positive whole number; the service enforces its upper bound. This is a per-launch choice rather than a request to change a benchmark-wide sampling profile. ## Reuse and deliberate repeats Run evaluation reopens a Run with matching immutable conditions when one already exists, including a queued, running, completed or failed Run. It does not start extra Trials merely because the action was selected again. Use **Run again** when another independent execution is intended. A repeat keeps the referenced conditions. Changed conditions must be handled explicitly rather than silently joining a different population. ## Object and state changes A new admission records one Run per selected candidate, its Trials, exact Harness and Benchmark Versions, execution allocation and evaluation settings. A matching admission returns the existing Run. Recorded responses, judgments and telemetry accumulate independently. The Run automatically selected as **Used for score** is chosen before results are known. Later repeats remain in Run history and do not silently replace it. ## Success criteria - The selected Versions, machine and Trial count match the intended experiment. - Every planned Case and Trial remains accounted for. - Missing judgments remain incomplete. - A repeat is distinguishable from reopening existing work. ## Common failure modes - Choosing a different saved Version than intended. - Mistaking output completion for completed evaluation. - Treating Run evaluation as an unconditional repeat. - Reading only successful Trials from an incomplete Run. ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings) - [Evaluation Results](/docs/benchmark-evaluations/results) - [Run Details and Metadata](/docs/benchmark-evaluations/run-details) {% /related-card-grid %} ## Related troubleshooting pages {% related-card-grid title="Related troubleshooting pages" %} - [Benchmark run troubleshooting](/docs/troubleshooting/benchmark-runs) - [Missing Outputs](/docs/troubleshooting/missing-outputs) {% /related-card-grid %}