# Evaluate Without Uploading Production Cases

Generated: 2026-11-13T08:01:26.948Z
Source build: local
Canonical docs: https://teammately.ai/docs

<doc id="security.synthetic-evaluation" slug="security/synthetic-evaluation" url="https://teammately.ai/docs/security/synthetic-evaluation.md" title="Evaluate Without Uploading Production Cases" updated="2026-11-09" status="stable">
---
id: security.synthetic-evaluation
title: Evaluate Without Uploading Production Cases
summary: Use synthetic Cases, fictional materials, and a test-only Harness environment to evaluate behavior while keeping actual production records outside the experiment.
kind: task
product_area: security
status: stable
updated: 2026-11-09
canonical: /docs/security/synthetic-evaluation
---

# Evaluate Without Uploading Production Cases

Uploading production Cases is optional. You can construct the situations your agent must handle from behavioral requirements, expert judgment, and coverage design, then evaluate them in Trialground. Actual customer conversations, transactions, and confidential documents can remain outside that experiment.

Lemon Weave provides the synthesis capability: it constructs candidate Cases, controlled alternatives, and supporting materials from the context and coverage you supply. Trialground runs saved Harness Versions in managed sandboxes and assesses recorded behavior against your benchmark criteria.

## Prerequisites

- A Project whose Project Agent Brief and coverage definitions contain only information permitted for this experiment.
- A selected Benchmark, its Project Input Schema, and permission to manage coverage and candidate membership.
- A named behavior to examine and the fictional facts or materials needed to examine it.
- A Harness with test-only endpoints and credential grants when actual production data must remain outside execution.

## Inspect the generation context

Describe the task, relevant rules, exceptions, and conditions using information approved for this workflow. Use fictional names, identifiers, documents, and tool results. Match the Project Input Schema and required material formats so the Case has the structure your candidate expects.

If a real incident motivates the Case, describe the behavioral distinction without copying its private records. Generated content can reproduce sensitive facts supplied in instructions or reference material. Review inputs and materials before admitting them to the benchmark.

Generation uses Project context as well as the text you enter for one request. Inspect the **Project Agent Brief**, the selected Dimensions, Topics and Scenarios, their descriptions and source context, and any selected sample Cases. Descriptions derived from an actual incident can contain private details even when no production Case is selected.

The fictional-only boundary applies to this whole input set. A request to use fictional names is a generation instruction; it is separate from excluding sensitive source information before processing.

## Prepare new Cases through Shots

1. Open the intended Benchmark's **Coverage Design → Canvas**, then choose **Shots**.
2. Start a **New Shot** and select the relevant conditions. Choose **Add** and set **Cases to add** to the amount needed for this experiment.
3. Enter the fictional situation and construction requirements in **Additional instructions**. Leave **Sample cases** empty, or retain only samples approved for this workflow. Remove any sample that should not enter generation.
4. Choose **Stage shot**. Inspect the staged request and preview. Use **Update staged shot** if you change it before preparation.
5. Choose **Prepare staged shots** after the preview is ready. This addition path requests newly synthesized candidates. Preparation through **Rules** can reuse existing Case Pool candidates, so source behavior should be read from the operation you actually choose.
6. Inspect the prepared Cases and required materials in the Canvas preview. Review their source relationships and eligibility, select the acceptable candidates, and approve them through the preview's review controls.
7. Confirm the current editable Benchmark membership. Create the intended Snapshot and Benchmark Version before using that fixed population and its criteria for evaluation.

Shots can be configured with sample Cases. Synthesis-only creates new candidates using the Project Agent Brief, selected facet context, and any supplied samples. Review each of these inputs for the experiment's information requirements.

## Keep the execution separate

Configure the Harness to use test-only services, fictional state, and credentials scoped to test resources. Review retrieval sources, environment configuration, callbacks, tool responses, and outputs so the whole experiment follows the intended information boundary.

Trialground manages the execution environment; the Harness defines model calls, tools, and task behavior. A job can process several Cases in one process and workspace. Configure per-Case state and reset behavior when independence is required. The selected endpoints, credential grants, and network configuration define the execution's connectivity.

Use [Network Controls and Isolation](/docs/security/network-isolation) to review workspace access and private ingestion or inference separately from the candidate's data access.

## Choose where execution runs

| Execution choice | What stays separate | What Teammately receives |
| --- | --- | --- |
| Managed Trialground execution with synthetic Cases and test-only tools | Actual production records and systems can remain outside the experiment | Supplied context, Cases, materials, the saved Harness, responses, and captured activity |
| Execution in your own environment with output submission | Your application execution and unsubmitted internal records remain there | Benchmark Cases and criteria already in Teammately, submitted responses or failures, and any included trajectories or telemetry |

External execution keeps the candidate runtime in your environment. Teammately receives the outputs and trajectories you submit for assessment. Review those submissions for the information they contain.

## Review what the assessment establishes

Experts should check realism, valid conditions, expected behavior, and the distinctions each Case exercises. Freeze the intended Case population and criteria in a Benchmark Version, then inspect responses and recorded steps.

Synthetic evaluations assess behavior in the constructed situations. Use representative production observations to understand event frequency, and assess actual integrations and production behavior within the customer's approved environment.

## Object and state changes

Staging a Shot records the proposed request in the current editing session. Preparation records an execution and produces candidates for review. Approval changes the editable Benchmark population through its owning action. Saved Snapshots and historical Runs retain their recorded populations and candidate identities.

## Success criteria

- Generation context, selected samples, and supporting materials meet the experiment's information boundary.
- Prepared additions are newly synthesized, and the accepted Cases demonstrate the intended distinction.
- The selected Harness uses only the endpoints, identities, and resources appropriate for the experiment.
- The resulting Benchmark Version, candidate Version, outputs, and captured activity are identifiable.

## Common failure modes

- Starting a synthesis-only request with sensitive Project Brief or facet content still in scope.
- Leaving an actual customer Case attached as a sample.
- Confusing a Rules preparation that reuses Cases with a new Shot addition.
- Connecting a fictional Case to a production retrieval source or credential.
- Treating preparation completion as approval or assuming execution teardown deletes retained outputs.

{% example-demo title="An invented refund history" %}
A team wants an agent to recognize that a partial refund has already been issued. It defines the rule with an approved fictional policy, creates an invented order with a $120 payment and a prior $40 refund, and supplies a test-only refund service. Variants change the earlier operation's status or repeat its request identifier. Reviewers check the eligible remainder, uncertain completion, and duplicate-action handling without supplying a customer's actual order or payment record.
{% /example-demo %}

## Related workflows

{% related-card-grid title="Construct and assess the experiment" %}
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Harnesses](/docs/assets/harnesses)
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Connect Model Outputs](/docs/integrations/connect-model-outputs)
- [Data Handling and AI Processing](/docs/security/data-handling)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Understand the experiment's boundaries" %}
- [Network Controls and Isolation](/docs/security/network-isolation)
- [Project Agent Brief](/docs/agent-setup/project-context)
- [Project Input Schema](/docs/project-settings/input-schema)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Investigate an unsuitable candidate" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
{% /related-card-grid %}
</doc>

<doc id="security.overview" slug="security" url="https://teammately.ai/docs/security.md" title="Security and Data" updated="2026-11-09" status="stable">
---
id: security.overview
title: Security and Data
summary: Understand Teammately's security controls, customer-data flows, and the option to evaluate with synthetic Cases without uploading production records.
kind: concept
product_area: security
status: stable
updated: 2026-11-09
canonical: /docs/security
---

# Security and Data

You can develop and evaluate AI in Teammately without uploading your production Cases, customer conversations, or business records. Use synthetic Cases and fictional supporting materials to exercise the behavior your team needs. Configure the candidate's tools and credentials for a separate test environment.

Teammately also supports workflows using customer-supplied content. The data you enter, connect, generate, execute, and submit determines what each workflow processes.

## Decision checkpoint

| Your question | Read | What it explains |
| --- | --- | --- |
| Can we keep actual records outside the experiment? | [Synthetic Evaluation](/docs/security/synthetic-evaluation) | Constructed Cases, test-only environments, and what to inspect |
| What reaches Teammately and model services? | [Data Handling and AI Processing](/docs/security/data-handling) | Inputs, outputs, providers, storage, and geographic scope |
| Who can access our work? | [Access Control](/docs/security/access-control) | Workspace, Project, reviewer, artifact, and credential boundaries |
| How do we restrict network access? | [Network Controls and Isolation](/docs/security/network-isolation) | IP restrictions, private ingestion and inference, and execution boundaries |
| Which actions can we inspect or export? | [Audit Logs and Activity](/docs/security/audit-activity) | Workspace events, filtered CSV export, and artifact-specific history |
| Which security commitments apply? | [Trust Center](https://teammately.ai/trust) | Published controls, assessment status, providers, and agreement requirements |
| What should procurement review? | [Security Review and Assurance](/docs/security/assurance) | Independent assessment status and workflow-specific requirements |
| How can we explain a result later? | [Reproducibility](/docs/governance/reproducibility) | The Cases, criteria, and candidate versions used for assessment |

## Security controls and commitments

| Protection | How it works | Scope |
| --- | --- | --- |
| Customer files and backups | Azure Blob Storage encryption at rest with Microsoft-managed keys; public Blob access disabled | Customer and PostgreSQL backup storage accounts |
| Database storage | Platform-managed encryption for database-cluster disks | Database-cluster storage |
| Workspace and Project access | Server-side permissions, including direct user and workspace-group Project grants | The selected workspace, Project, and action |
| Protected files and previews | Authenticated workspace, Project access, and deletion checks | Access requires an authenticated, authorized request |
| Network access | Workspace IP restrictions and deployment-specific private connectivity | Inbound app access, ingestion, and model inference have distinct controls |
| Runtime credentials | Separate workspace secrets and explicit Harness grants | The selected Harness and its authorized execution context |
| Reviewable activity | Workspace Audit Log, artifact history, and recorded execution activity | Each surface records its own event types and scope |
| Recovery | Backup procedures, storage versioning, and soft deletion | Recovery protection and permanent erasure have different lifecycles |

The [Trust Center](https://teammately.ai/trust) gives the individual control descriptions, assessment methods, and dates. Core hosting location, model-processing geography, and traffic delivery are separate considerations.

Teammately does not use customer workspace data to train shared generative models without express written agreement. Providing a requested AI service can still involve processing and retaining content. Review the selected workflow and provider terms separately.

## Compliance and procurement

Use the Trust Center for Teammately's assessment and audit status, and the applicable agreement for contractual commitments. Each assessment identifies the organization and services it covers. Review the controls and agreement that apply to your intended workflows.

Teammately's security program follows the SOC 2 Security criteria, with controls assessed internally. An independent SOC 2 Type II audit is in progress. [Security Review and Assurance](/docs/security/assurance) explains how to review the assessment status and the scope of your intended workflows.

HIPAA-compliant deployments require a signed Business Associate Agreement and deployment approval before protected health information is submitted. The approved services, workflows, integrations, and data categories define that scope.

Retention, deletion, recovery, and processing requirements should cover the intended workflow, including model-service records, logs, files, and backups. A recovery window is separate from a permanent-erasure timeline.

## Related guides

{% related-card-grid title="Understand your data flow" %}
- [Data Handling and AI Processing](/docs/security/data-handling)
- [Access Control](/docs/security/access-control)
- [Synthetic Evaluation](/docs/security/synthetic-evaluation)
- [Network Controls and Isolation](/docs/security/network-isolation)
- [Audit Logs and Activity](/docs/security/audit-activity)
- [Security Review and Assurance](/docs/security/assurance)
{% /related-card-grid %}
</doc>

<doc id="security.data-handling" slug="security/data-handling" url="https://teammately.ai/docs/security/data-handling.md" title="Data handling" updated="2026-11-11" status="stable">
---
id: security.data-handling
title: Data handling
summary: Trace what enters Teammately, what AI services process, and how workflow choices affect storage, processing locations, retention, and deletion.
kind: reference
product_area: security
status: stable
updated: 2026-11-11
canonical: /docs/security/data-handling
---

# Data handling

## Definition

Customer content includes more than uploaded Cases. Project instructions, Reference Materials, expert contributions, files, candidate responses, and captured tool activity can all contain business information. Synthetic experiments process and retain the selected context, constructed inputs, and resulting activity according to the chosen workflow.

## Fields, states, or lifecycle rules

| Workflow | Information processed | What to review |
| --- | --- | --- |
| Project setup and expert work | Entered instructions, materials, comments, answers, and supported recordings | Which information participants supply and who can access it |
| Case synthesis | Coverage requirements, supplied context, generation instructions, and generated Cases or materials | Whether every source is suitable for the intended processing |
| Managed Trialground execution | Selected Case inputs, materials, saved Harness, outputs, and captured trajectory events | Tools, granted credentials, model calls, returned data, and retained records |
| External execution and output submission | Benchmark Cases and criteria held in Teammately, plus submitted responses, failures, and any included trajectories | What remains in your environment and what the submission transmits |
| Connected integrations | Data selected for the specific connection and operation | Provider, access scope, processing location, and applicable terms |

### Follow the information through the workflow

For a synthetic experiment, permitted Project context and fictional requirements feed Case and material generation. The selected Cases, materials and saved Harness then enter managed execution, or your own execution environment. Submitted responses and captured activity enter assessment and the resulting Run records.

| Information | Where it enters or remains | Selection and access responsibility |
| --- | --- | --- |
| Project Agent Brief and coverage definitions | Context for construction and other selected AI workflows | Review the Brief and relevant Dimensions, Topics, Scenarios, and source descriptions |
| Sample Cases and source material | Generation context when the selected operation includes them | Remove unapproved samples before preparation; new synthesis and source exclusion are different choices |
| Inputs, generated materials and candidate code | Benchmark assets and the selected execution | Project permissions and candidate selection govern use |
| Tool requests and returned content | The Harness's configured services and supported captured activity | Endpoints, resource permissions, credential grants, and capture configuration |
| Responses and recorded execution | Run, Results, and supported execution records | Review content before external submission and apply the intended access scope |
| AI-service state | The model feature requested by the owning workflow | Provider terms, feature configuration, processing location, and retention scope |

Private ingestion and inference provide private transport for agreed paths. Review the submitted information and each service's processing location alongside that network configuration. See [Network Controls and Isolation](/docs/security/network-isolation).

### Core services and processing locations

The [Trust Center](https://teammately.ai/trust) lists Microsoft Azure for application hosting, databases, storage, backups, and AI processing; Auth0 for authentication; and Cloudflare for proxied traffic and delivery. Core Azure hosting is in Central US. Model services and traffic delivery have their own processing geographies, described below and in the Trust Center.

Teammately's platform AI processing uses Azure OpenAI or Microsoft Foundry. Global model deployments may process prompts and responses across supported geographies. Customer-selected Harness endpoints and integrations have their own processing scope.

### Model training and service storage

Teammately does not use customer workspace data to train shared generative models without the customer's express written agreement. This commitment is separate from processing content for a requested service or retaining operational records. Customer-requested weights training is a separate workflow with its own selected dataset and provider.

Model-service storage follows the selected API feature and workflow. Managed execution retains outputs and captured trajectory events outside the sandbox, where they remain available after execution ends under the applicable retention arrangements. Review [Microsoft's AI data-processing terms](https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy) alongside the features and integrations you select.

### Retention, deletion, and recovery

Your requirements should distinguish active application records, files, model-service data, logs, and backups. The applicable agreement and selected workflows determine the retention and deletion commitments. Storage soft deletion and backup recovery periods describe recovery protection. Permanent-erasure timelines apply to the agreed records and stores in the deletion scope.

The [Privacy Policy](https://teammately.ai/privacy) and Trust Center describe privacy-request handling. Customer-controlled workspace content is handled on the customer's documented instructions, with identity and authority checks where appropriate.

| Record category | Retention or deletion scope to establish |
| --- | --- |
| Active Project and Benchmark records | The selected objects, permitted use, historical versions, and applicable agreement |
| Files, generated artifacts and supported recordings | Originals, derivatives, versions, and references held by the workflow |
| Model-service records | Uploaded content and stored state requested by the selected feature |
| Application and audit logs | The relevant event collection and its retention; infrastructure-log retention is separate |
| Backups and recovery copies | Recovery windows, expiration, and the handling of previously deleted content after restoration |

Use the privacy-request process to identify the authorized objects and stores in a deletion request. Record the agreed completion criteria and treatment of historical or recovery copies. Review active records, retained execution data, and recovery copies as distinct parts of that scope.

### Recovery objectives

Teammately's published controls describe database backup and restoration procedures, storage versioning, and soft-deletion protections. Review those protections alongside the required recovery point and recovery time, then define the applicable objectives for the agreed deployment and workflows.

## Related task pages

{% related-card-grid title="Choose the intended data flow" %}
- [Synthetic Evaluation](/docs/security/synthetic-evaluation)
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Connect Model Outputs](/docs/integrations/connect-model-outputs)
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
{% /related-card-grid %}
</doc>

<doc id="concepts.weave" slug="concepts/weave" url="https://teammately.ai/docs/concepts/weave.md" title="Lemon Weave" updated="2026-11-09" status="stable">
---
id: concepts.weave
title: Lemon Weave
summary: Construct Cases, controlled alternatives, supporting materials, and targeted training examples from coverage needs and expert judgment.
kind: concept
product_area: weave
status: stable
updated: 2026-11-09
canonical: /docs/concepts/weave
---

# Lemon Weave

A useful synthetic Case makes a missing distinction observable. Lemon Weave constructs situations around the conditions, exceptions, and tradeoffs your benchmark needs to examine.

Coverage supplies the purpose. Selected ontology tuples describe the conditions that should occur together; the Case and its materials give those conditions a concrete form.

## Definition

Lemon Weave constructs candidate Cases, controlled response alternatives, and supporting artifacts from project context and coverage design. Cases follow **Project Input Schema**, including required inputs and accepted material formats.

Worlds describe environment state and capabilities for tasks that unfold through action. Static documents, images, workbooks, source files, and other supporting content travel with the Case as materials. The configured experiment determines how the Harness uses its environment.

The same coverage foundation guides targeted dataset augmentation for weights training.

## Construct Cases without production records

You do not need to upload actual customer Cases to use synthesis. Start with approved requirements and expert descriptions of the situations that matter, then construct fictional inputs and supporting materials. Review the supplied context as well as the generated Case: copying a sensitive record into generation instructions still introduces that record to the workflow.

For evaluation, pair those Cases with a Harness configured for test-only tools and credentials. [Evaluate Without Uploading Production Cases](/docs/security/synthetic-evaluation) explains this separation and what synthetic results establish.

## Decision checkpoint

| Your need | Start with | Inspect |
| --- | --- | --- |
| Exercise a missing combination | Coverage Design → Canvas | The tuple, Case, and what it demonstrates |
| Reuse a real example | Assets → Cases and Benchmark Datasets | Input, materials, and intended membership |
| Examine a tradeoff | Comparison Directions and Expert Contributions | Controlled alternatives and expert reasoning |
| Supply supporting artifacts | Project Input Schema and Case materials | Accepted formats and required context |
| Teach a recurring failure | Weights training | Demonstrations or preferences, splits, and coverage |

## Make the difficulty meaningful

A challenging Case can combine ordinary conditions in a consequential way: an urgent request, an outdated source, and an exception requiring approval. Each condition should matter to the behavior being assessed.

Review whether the input and materials make the situation understandable. Inspect the controlling facts, ambiguity, and source relationships. Select Cases that add the intended coverage to the Benchmark.

## Develop teaching material from the same distinction

An expert rubric can identify what a demonstration should teach. Ontology tuples guide variation across examples. Preference data can contrast approaches where the expert’s reasoning explains the better choice.

In **Weights training**, inspect Dataset and Coverage, prepare an exact download, or configure a connected training service. Dataset construction and training execution are separate operations. The selected training environment and agreement determine training compute and execution scope. Keep separate assessment Cases for judging whether the trained candidate carries the behavior beyond its teaching material.

## Related workflows

{% related-card-grid title="Construct and review material" %}
- [Coverage Design](/docs/coverage-design)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Weights Training](/docs/weights-training)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Connect the context" %}
- [Project Input Schema](/docs/project-settings/input-schema)
- [Comparison Directions](/docs/assets/comparison-directions)
- [Worlds](/docs/assets/worlds)
{% /related-card-grid %}
</doc>

<doc id="concepts.trialground" slug="concepts/trialground" url="https://teammately.ai/docs/concepts/trialground.md" title="Trialground" updated="2026-11-09" status="stable">
---
id: concepts.trialground
title: Trialground
summary: Execute saved Harnesses in managed sandboxes and assess recorded behavior against expert rubrics.
kind: concept
product_area: trialground
status: stable
updated: 2026-11-09
canonical: /docs/concepts/trialground
---

# Trialground

Trialground connects Harness execution with rubric evaluation. Your engineers can try candidate implementations across the situations in a benchmark and inspect how they behaved.

A change to retrieval, instructions, tool use, or orchestration becomes a concrete experiment with recorded inputs, responses, and judgments.

## Definition

Managed execution runs an exact saved Python Harness Version with its declared dependencies and selected benchmark inputs. Trialground prepares the dependency environment, manages execution, and retains outputs and captured trajectory events outside the sandbox.

**Benchmark Evaluations** provides **Runs**, **Results**, and **Dashboard**. A Run contains its planned Trials; each Trial exercises the candidate across the recorded Case population.

You can also submit externally executed responses, failures, and supported trajectories for assessment against the selected benchmark.

## Decision checkpoint

| Your question | Open | What to inspect |
| --- | --- | --- |
| Which candidate will execute? | Runs | Saved Version, machine, and Trial count |
| What did the agent do? | Results | Response, captured trajectory, and capture status |
| Why did it receive that score? | Results | Applicable rubrics, points, penalties, and gates |
| How do candidates compare? | Results and Dashboard | Comparable settings and complete populations |
| What happened across repetitions? | Run history | Independent Runs and the Run used for score |

## Prepare and run the experiment

Publish the intended Harness Version and grant the credentials it needs. Declared Python dependencies form the prepared execution image. Select the machine and Trials when launching an evaluation.

The Harness defines model calls, tools, task behavior, and state handling. An execution job can process several inputs in one process and workspace. Configure per-Case state and World reset behavior as part of the experiment.

Use synthetic Cases, fictional materials, and test-only endpoints and credentials to keep actual production records outside an experiment. A managed sandbox does not itself prevent a Harness from retrieving data through its configured tools. Outputs and captured activity remain part of the Run after execution ends. [Synthetic Evaluation](/docs/security/synthetic-evaluation) and [Data Handling and AI Processing](/docs/security/data-handling) explain these choices.

Use a saved-Harness debug run to try supplied inputs during development. Use a Benchmark Version to assess the candidate against a fixed Case population and criteria.

## Read recorded behavior and judgments

Captured trajectories make instrumented actions and tool use inspectable. Capture status identifies the recorded material available for assessment. Preserve any other execution files through the configured persistence path when the task needs them.

From **Results**, choose **Inspect recorded steps** for a Case or an individual Trial judgment. Recorded model requests and responses, tool arguments and results, and agent delegations retain their execution references. Supported instrumentation determines which actions are captured. Partial, unsupported, failed and unknown capture remain explicit; missing capture does not establish that an action was omitted.

Trajectory criteria assess recorded behavior. Response criteria assess the submitted answer. A saved judgment observation view preserves what its evaluator received, while the associated execution record supports investigation of the candidate. These records are separate from Lemon Goal's session narration and private model reasoning.

Rubric judgments feed the project Regime. Positive points reward passing conditions; penalties deduct for failures; mandatory gates express requirements that must pass. Incomplete judgments remain visible across the full population.

## Investigate the next change

Run evaluation reopens matching existing work. Use **Run again** for an independent repetition. **Used for score** identifies the Run chosen before its results were known.

Follow a failure into a Harness change, a coverage question, or clarification of the standard. Engineers can compare the proposed intervention against the same benchmark that motivated it.

## Related workflows

{% related-card-grid title="Run and inspect a candidate" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Import External Evaluation Outputs](/docs/benchmark-evaluations/output-mapping)
{% /related-card-grid %}

## Related reference pages

{% related-card-grid title="Understand the experiment" %}
- [Harnesses](/docs/assets/harnesses)
- [Rubric Points and Benchmark Scores](/docs/correctness-governance/scoring)
- [Run Details and Metadata](/docs/benchmark-evaluations/run-details)
{% /related-card-grid %}
</doc>

<doc id="coverage.synthesize-cases" slug="coverage-engineering/synthesize-cases" url="https://teammately.ai/docs/coverage-engineering/synthesize-cases.md" title="Synthesize Cases with Lemon Weave" updated="2026-11-11" status="stable">
---
id: coverage.synthesize-cases
title: Synthesize Cases with Lemon Weave
summary: Generate benchmark-scoped input candidates from explicit coverage needs and the project input/material contract.
kind: task
product_area: coverage_engineering
status: stable
updated: 2026-11-11
canonical: /docs/coverage-engineering/synthesize-cases
---

# Synthesize Cases with Lemon Weave

Synthesize when a named input situation is missing and its expected behavior is clear enough to review.

Production Case uploads are optional. You can supply approved requirements and fictional source context, then generate the input and materials needed for the situation. For the complete data and execution boundary, read [Evaluate Without Uploading Production Cases](/docs/security/synthetic-evaluation).

## Prerequisites

- An explicit coverage requirement or tuple.
- Relevant project facets and source context.
- A Project Input Schema that represents the intended input and materials.
- A reviewer who can assess realism and fit.

## Steps

1. Open the intended Benchmark's Coverage Design → Canvas.
2. Identify the missing situation and inspect existing Cases before generating more.
3. For newly synthesized additions, open **Shots**, start a **New Shot**, choose **Add**, and set **Cases to add**.
4. Select the conditions and supply **Additional instructions**. Inspect the Project Agent Brief, facet context and any **Sample cases** that generation will receive.
5. Choose **Stage shot**, review the preview, then choose **Prepare staged shots** when it is ready.
6. Inspect prepared candidates and required materials in the Canvas preview, approve acceptable Cases, and check current membership and the remaining requirement.
7. Create a new Snapshot when the changed population should define a later evaluation.

Use **Assets → Cases** to import real examples, connect integrations, or create a reusable Case manually.

## Input and material authority

Project Input Schema supplies allowed artifact families, accepted extensions and required material keys. A coverage tuple can request an allowed artifact but cannot override that contract.

Keep retrieved and synthesized source relationships visible. A polished response or a completed material job does not establish that the input demonstrates its intended situation.

Shot additions request new synthesis. Rules preparation can reuse existing Cases while filling its remaining need. Synthesis can use supplied sample Cases and Project context in either workflow. For an experiment that excludes actual records, use the source and context review in [Evaluate Without Uploading Production Cases](/docs/security/synthetic-evaluation).

Cases generated for an Inbox observation are staged as separate items. Their creation does not automatically adopt them or resolve the observation.

## Object and state changes

Generation records accepted planning/input settings and prepares candidates. Review and adoption determine editable membership. Historical Snapshots and Runs stay unchanged.

## Success criteria

- Each candidate demonstrates a named input need.
- The input is realistic and conformant.
- Required materials are available and inspectable.
- Source provenance and adoption state remain distinguishable.

## Common failure modes

- Asking for volume without a coverage purpose.
- Generating around an unresolved correctness standard.
- Assuming accepted extensions require every format.
- Counting generation completion as approved membership.
- Preparing Cases without checking the selected Benchmark and coverage need.
## Related reference pages

{% related-card-grid title="Related reference pages" %}
- [Coverage Design](/docs/coverage-design)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Input Coverage Inbox](/docs/coverage-design/inbox)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}

## Related troubleshooting pages

{% related-card-grid title="Related troubleshooting pages" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
{% /related-card-grid %}
</doc>