Evaluate Without Uploading Production Cases
Uploading production Cases is optional. You can construct the situations your agent must handle from behavioral requirements, expert judgment, and coverage design, then evaluate them in Trialground. Actual customer conversations, transactions, and confidential documents can remain outside that experiment.
Lemon Weave provides the synthesis capability: it constructs candidate Cases, controlled alternatives, and supporting materials from the context and coverage you supply. Trialground runs saved Harness Versions in managed sandboxes and assesses recorded behavior against your benchmark criteria.
Prerequisites
- A Project whose Project Agent Brief and coverage definitions contain only information permitted for this experiment.
- A selected Benchmark, its Project Input Schema, and permission to manage coverage and candidate membership.
- A named behavior to examine and the fictional facts or materials needed to examine it.
- A Harness with test-only endpoints and credential grants when actual production data must remain outside execution.
Inspect the generation context
Describe the task, relevant rules, exceptions, and conditions using information approved for this workflow. Use fictional names, identifiers, documents, and tool results. Match the Project Input Schema and required material formats so the Case has the structure your candidate expects.
If a real incident motivates the Case, describe the behavioral distinction without copying its private records. Generated content can reproduce sensitive facts supplied in instructions or reference material. Review inputs and materials before admitting them to the benchmark.
Generation uses Project context as well as the text you enter for one request. Inspect the Project Agent Brief, the selected Dimensions, Topics and Scenarios, their descriptions and source context, and any selected sample Cases. Descriptions derived from an actual incident can contain private details even when no production Case is selected.
The fictional-only boundary applies to this whole input set. A request to use fictional names is a generation instruction; it is separate from excluding sensitive source information before processing.
Prepare new Cases through Shots
- Open the intended Benchmark's Coverage Design → Canvas, then choose Shots.
- Start a New Shot and select the relevant conditions. Choose Add and set Cases to add to the amount needed for this experiment.
- Enter the fictional situation and construction requirements in Additional instructions. Leave Sample cases empty, or retain only samples approved for this workflow. Remove any sample that should not enter generation.
- Choose Stage shot. Inspect the staged request and preview. Use Update staged shot if you change it before preparation.
- Choose Prepare staged shots after the preview is ready. This addition path requests newly synthesized candidates. Preparation through Rules can reuse existing Case Pool candidates, so source behavior should be read from the operation you actually choose.
- Inspect the prepared Cases and required materials in the Canvas preview. Review their source relationships and eligibility, select the acceptable candidates, and approve them through the preview's review controls.
- Confirm the current editable Benchmark membership. Create the intended Snapshot and Benchmark Version before using that fixed population and its criteria for evaluation.
Shots can be configured with sample Cases. Synthesis-only creates new candidates using the Project Agent Brief, selected facet context, and any supplied samples. Review each of these inputs for the experiment's information requirements.
Keep the execution separate
Configure the Harness to use test-only services, fictional state, and credentials scoped to test resources. Review retrieval sources, environment configuration, callbacks, tool responses, and outputs so the whole experiment follows the intended information boundary.
Trialground manages the execution environment; the Harness defines model calls, tools, and task behavior. A job can process several Cases in one process and workspace. Configure per-Case state and reset behavior when independence is required. The selected endpoints, credential grants, and network configuration define the execution's connectivity.
Use Network Controls and Isolation to review workspace access and private ingestion or inference separately from the candidate's data access.
Choose where execution runs
| Execution choice | What stays separate | What Teammately receives |
|---|---|---|
| Managed Trialground execution with synthetic Cases and test-only tools | Actual production records and systems can remain outside the experiment | Supplied context, Cases, materials, the saved Harness, responses, and captured activity |
| Execution in your own environment with output submission | Your application execution and unsubmitted internal records remain there | Benchmark Cases and criteria already in Teammately, submitted responses or failures, and any included trajectories or telemetry |
External execution keeps the candidate runtime in your environment. Teammately receives the outputs and trajectories you submit for assessment. Review those submissions for the information they contain.
Review what the assessment establishes
Experts should check realism, valid conditions, expected behavior, and the distinctions each Case exercises. Freeze the intended Case population and criteria in a Benchmark Version, then inspect responses and recorded steps.
Synthetic evaluations assess behavior in the constructed situations. Use representative production observations to understand event frequency, and assess actual integrations and production behavior within the customer's approved environment.
Object and state changes
Staging a Shot records the proposed request in the current editing session. Preparation records an execution and produces candidates for review. Approval changes the editable Benchmark population through its owning action. Saved Snapshots and historical Runs retain their recorded populations and candidate identities.
Success criteria
- Generation context, selected samples, and supporting materials meet the experiment's information boundary.
- Prepared additions are newly synthesized, and the accepted Cases demonstrate the intended distinction.
- The selected Harness uses only the endpoints, identities, and resources appropriate for the experiment.
- The resulting Benchmark Version, candidate Version, outputs, and captured activity are identifiable.
Common failure modes
- Starting a synthesis-only request with sensitive Project Brief or facet content still in scope.
- Leaving an actual customer Case attached as a sample.
- Confusing a Rules preparation that reuses Cases with a new Shot addition.
- Connecting a fictional Case to a production retrieval source or credential.
- Treating preparation completion as approval or assuming execution teardown deletes retained outputs.
Worked example
An invented refund history
A team wants an agent to recognize that a partial refund has already been issued. It defines the rule with an approved fictional policy, creates an invented order with a $120 payment and a prior $40 refund, and supplies a test-only refund service. Variants change the earlier operation's status or repeat its request identifier. Reviewers check the eligible remainder, uncertain completion, and duplicate-action handling without supplying a customer's actual order or payment record.