# Security and Data
Generated: 2026-11-13T08:01:21.760Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: security.overview
title: Security and Data
summary: Understand Teammately's security controls, customer-data flows, and the option to evaluate with synthetic Cases without uploading production records.
kind: concept
product_area: security
status: stable
updated: 2026-11-09
canonical: /docs/security
---
# Security and Data
You can develop and evaluate AI in Teammately without uploading your production Cases, customer conversations, or business records. Use synthetic Cases and fictional supporting materials to exercise the behavior your team needs. Configure the candidate's tools and credentials for a separate test environment.
Teammately also supports workflows using customer-supplied content. The data you enter, connect, generate, execute, and submit determines what each workflow processes.
## Decision checkpoint
| Your question | Read | What it explains |
| --- | --- | --- |
| Can we keep actual records outside the experiment? | [Synthetic Evaluation](/docs/security/synthetic-evaluation) | Constructed Cases, test-only environments, and what to inspect |
| What reaches Teammately and model services? | [Data Handling and AI Processing](/docs/security/data-handling) | Inputs, outputs, providers, storage, and geographic scope |
| Who can access our work? | [Access Control](/docs/security/access-control) | Workspace, Project, reviewer, artifact, and credential boundaries |
| How do we restrict network access? | [Network Controls and Isolation](/docs/security/network-isolation) | IP restrictions, private ingestion and inference, and execution boundaries |
| Which actions can we inspect or export? | [Audit Logs and Activity](/docs/security/audit-activity) | Workspace events, filtered CSV export, and artifact-specific history |
| Which security commitments apply? | [Trust Center](https://teammately.ai/trust) | Published controls, assessment status, providers, and agreement requirements |
| What should procurement review? | [Security Review and Assurance](/docs/security/assurance) | Independent assessment status and workflow-specific requirements |
| How can we explain a result later? | [Reproducibility](/docs/governance/reproducibility) | The Cases, criteria, and candidate versions used for assessment |
## Security controls and commitments
| Protection | How it works | Scope |
| --- | --- | --- |
| Customer files and backups | Azure Blob Storage encryption at rest with Microsoft-managed keys; public Blob access disabled | Customer and PostgreSQL backup storage accounts |
| Database storage | Platform-managed encryption for database-cluster disks | Database-cluster storage |
| Workspace and Project access | Server-side permissions, including direct user and workspace-group Project grants | The selected workspace, Project, and action |
| Protected files and previews | Authenticated workspace, Project access, and deletion checks | Access requires an authenticated, authorized request |
| Network access | Workspace IP restrictions and deployment-specific private connectivity | Inbound app access, ingestion, and model inference have distinct controls |
| Runtime credentials | Separate workspace secrets and explicit Harness grants | The selected Harness and its authorized execution context |
| Reviewable activity | Workspace Audit Log, artifact history, and recorded execution activity | Each surface records its own event types and scope |
| Recovery | Backup procedures, storage versioning, and soft deletion | Recovery protection and permanent erasure have different lifecycles |
The [Trust Center](https://teammately.ai/trust) gives the individual control descriptions, assessment methods, and dates. Core hosting location, model-processing geography, and traffic delivery are separate considerations.
Teammately does not use customer workspace data to train shared generative models without express written agreement. Providing a requested AI service can still involve processing and retaining content. Review the selected workflow and provider terms separately.
## Compliance and procurement
Use the Trust Center for Teammately's assessment and audit status, and the applicable agreement for contractual commitments. Each assessment identifies the organization and services it covers. Review the controls and agreement that apply to your intended workflows.
Teammately's security program follows the SOC 2 Security criteria, with controls assessed internally. An independent SOC 2 Type II audit is in progress. [Security Review and Assurance](/docs/security/assurance) explains how to review the assessment status and the scope of your intended workflows.
HIPAA-compliant deployments require a signed Business Associate Agreement and deployment approval before protected health information is submitted. The approved services, workflows, integrations, and data categories define that scope.
Retention, deletion, recovery, and processing requirements should cover the intended workflow, including model-service records, logs, files, and backups. A recovery window is separate from a permanent-erasure timeline.
## Related guides
{% related-card-grid title="Understand your data flow" %}
- [Data Handling and AI Processing](/docs/security/data-handling)
- [Access Control](/docs/security/access-control)
- [Synthetic Evaluation](/docs/security/synthetic-evaluation)
- [Network Controls and Isolation](/docs/security/network-isolation)
- [Audit Logs and Activity](/docs/security/audit-activity)
- [Security Review and Assurance](/docs/security/assurance)
{% /related-card-grid %}
---
id: security.access-control
title: Access Control
summary: Understand workspace and Project access, reviewer assignments, protected artifacts, and credentials granted to evaluation Harnesses.
kind: reference
product_area: security
status: stable
updated: 2026-11-09
canonical: /docs/security/access-control
---
# Access Control
## Definition
Authentication identifies the account. Workspace and Project permissions determine access to work. Reviewer assignments identify expert tasks. Harness secret grants authorize runtime credential use. These controls serve different purposes and should be reviewed separately.
## Fields, states, or lifecycle rules
| Access need | Owning control | What the control establishes |
| --- | --- | --- |
| Sign in to the intended workspace | The organization's configured authentication flow | Account identity and workspace context |
| Assign workspace responsibilities | Admin → Directory → Members | Workspace role assignments; separate from direct Project membership |
| Admit members from approved domains | Admin → Security → Domain Control | Domain restrictions on invitations when enabled |
| Restrict incoming workspace access | Admin → Security → IP Address Control | Enabled, nonempty IP allowlist at authenticated workspace access |
| Work with a Project | Project permissions assigned to users or workspace groups | The actions authorized for the selected Project |
| Complete assigned expert work | Contribution and reviewer assignment | The scoped reviewer work; separate from workspace administration |
| Read files, previews or thumbnails | Artifact authorization | Workspace and Project access plus deletion status |
| Use a credential from a Harness | Project secret access and the Harness grant | Runtime resolution for the identified candidate and execution context |
| Inspect workspace audit events | Audit Log read capability | Authorized workspace event access |
Required capabilities are action-specific. For example, updating Domain Control or IP Address Control requires `users:manage`, while workspace Audit Log access requires `activitylogs:read_all`. Check the assigned capabilities and the owning workflow's approval permissions for the action you intend to perform.
- Project permissions can be assigned to individual users or workspace groups. Check direct grants, group membership, and workspace roles when determining effective access.
- **Admin → Directory → Members** exposes workspace role assignments. **Project Settings → Project Members** exposes individual Project memberships.
- A reviewer being assigned a task is distinct from administering the Project or accessing every Project artifact.
- File, preview, and thumbnail access requires an authenticated request with the appropriate workspace and Project permissions, and a check of the artifact's deletion status.
- Workspace secrets are managed separately from Harness source. Management responses show metadata and masked hints. Runtime credential resolution checks the execution context, expiration, and active grants.
- Personnel infrastructure access is distinct from application permission checks. The security policy requires an authorized business purpose for personnel access to customer content.
### Identity and invitations
Sign-in uses the configured authentication flow for the intended organization. Review the enabled sign-in methods, MFA policy, account lifecycle, and session settings against your identity requirements as part of [Security Review and Assurance](/docs/security/assurance).
In **Admin → Security → Domain Control**, add the approved email domains, enable the invitation restriction, and save it. The control applies to invitations. Review existing Members and their roles separately when changing who should retain access.
### Credentials and the execution boundary
Grant a Harness only the credentials its selected experiment needs. For an experiment that keeps actual production data separate, use test-only identities and endpoints with no access to production records. Keep secret values out of Case content, source files, Run Metadata, and outputs.
The saved Harness, connected endpoints, and granted access determine what tools can retrieve alongside the synthetic Case. Review returned content and captured tool arguments or results as part of the experiment.
Runtime grants are tied to the selected candidate and execution context. A credential for one customer's endpoint is not replaced by a Teammately platform credential when that endpoint rejects it. Configure the matching endpoint and test-only customer credential together.
The [Permissions reference](/docs/reference/permissions) and object-specific guides explain operating access. The [Trust Center](https://teammately.ai/trust) describes the published security controls and their individual scope.
## Related task pages
{% related-card-grid title="Inspect the relevant access" %}
- [Reviewer and Project Access](/docs/governance/reviewer-and-project-access)
- [Permissions Troubleshooting](/docs/troubleshooting/permissions)
- [Harnesses](/docs/assets/harnesses)
- [Synthetic Evaluation](/docs/security/synthetic-evaluation)
- [Network Controls and Isolation](/docs/security/network-isolation)
- [Audit Logs and Activity](/docs/security/audit-activity)
{% /related-card-grid %}
---
id: security.assurance
title: Security assurances
summary: Review Teammately's current assessment status and define the identity, processing, retention, recovery, and regulatory requirements of your intended workflows.
kind: reference
product_area: security
status: stable
updated: 2026-11-11
canonical: /docs/security/assurance
---
# Security assurances
## Definition
A security review connects your intended use with the applicable technical controls, operating responsibilities, provider terms, and agreements. Identify the workflows and information involved before treating a general security statement as a commitment for every operation.
## Fields, states, or lifecycle rules
### Assessment status
Teammately's security program follows the SOC 2 Security criteria, with controls assessed internally. An independent SOC 2 Type II audit is in progress. The [Trust Center](https://teammately.ai/trust) publishes the current status and gives the method and date for individual control descriptions.
Internal control review, adopted security policies, and a completed independent assessment are different forms of assurance. When reviewing a report, check the legal entity, included services, assessment period, exclusions, and customer responsibilities. Hosting and model-provider reports describe the provider controls within their stated scope; Teammately's assessment covers its own included services and workflows.
### Define the intended use
| Review topic | Establish for the selected workflows |
| --- | --- |
| Information submitted | Cases, Project instructions, Reference Materials, expert content, files, outputs, and recorded activity; whether production records are excluded |
| Identity | Required sign-in or SSO method, MFA enforcement, account provisioning and removal, and session requirements for the agreed identity setup |
| Authorization | Workspace roles, direct and group Project grants, reviewer assignments, artifact access, API credentials, and Harness grants |
| Network | Inbound IP restrictions, private ingestion, private model inference, endpoints, and network owners |
| Processing location | Core hosting, the selected model deployment, traffic delivery, and connected service locations |
| Provider processing | Data sent to each service, requested model features, stateful storage, and the applicable processing and training terms |
| Audit | Required event types, authorized readers, review periods, export needs, and retention scope |
| Retention and deletion | Active records, files, recordings, model-service state, logs, backups, and the treatment of deleted content after restoration |
| Recovery | Required recovery point and recovery time, backup scope, restoration responsibilities, and the workflows the commitment covers |
| Personnel and support | Authorized business purposes for content access and the responsibilities of your administrators and participating experts |
Review the sign-in methods, MFA policy, provisioning integrations, and session settings configured for your deployment against your organization's identity requirements. Domain restrictions govern which email domains are permitted for invitations.
### Regulated information
HIPAA-compliant deployments are available under a signed Business Associate Agreement and deployment approval. Complete both before submitting protected health information. The agreement identifies the approved services, workflows, data categories, integrations, providers, and responsibilities. PHI processing follows this approved deployment and agreement scope.
The Privacy Policy describes website, account, and business-administration processing. Customer workspace content is processed on the customer's instructions. The purpose and applicable agreement determine the processing role.
### Prepare a review request
Use the [Trust Center](https://teammately.ai/trust) contact destination to share your required workflows and controls. Include the categories of information and required locations, access, audit, retention, and recovery terms. Keep actual customer records, credentials, and regulated information out of the initial inquiry.
Record the agreed scope with your organization. Technical settings, contractual requirements, and independent assessment status should remain identifiable separately as the use changes.
## Related task pages
{% related-card-grid title="Review the relevant controls" %}
- [Data Handling and AI Processing](/docs/security/data-handling)
- [Access Control](/docs/security/access-control)
- [Network Controls and Isolation](/docs/security/network-isolation)
- [Audit Logs and Activity](/docs/security/audit-activity)
- [Evaluate Without Uploading Production Cases](/docs/security/synthetic-evaluation)
{% /related-card-grid %}
---
id: security.audit-activity
title: Activity history
summary: Inspect workspace activity, export filtered audit records, and distinguish those records from approval history, runtime credential events, and captured execution.
kind: reference
product_area: security
status: stable
updated: 2026-11-11
canonical: /docs/security/audit-activity
---
# Activity history
## Definition
The workspace Audit Log records supported application events with their actor and action context. An artifact's history records its own changes and decisions. Trialground records candidate behavior captured by supported instrumentation. Use the surface that owns the question you are investigating.
## Fields, states, or lifecycle rules
| Question | Surface | Scope |
| --- | --- | --- |
| Who performed a recorded workspace action? | Admin → Insights → Audit Log | Events available to the authorized workspace reader |
| Who approved or changed a Policy or other governed artifact? | The artifact's activity or approval history | The particular object and version |
| What did this candidate do? | Benchmark Evaluations → Results and recorded steps | Captured requests, responses, tool activity, and capture status |
| Which credential resolution was allowed or denied? | Runtime credential-access records | The credential service's execution context; separate from universal audit coverage |
### Inspect workspace events
Open **Admin → Insights → Audit Log**. Workspace audit access requires the `activitylogs:read_all` capability, granted separately from sign-in access and expert assignments.
Choose the date range and use the available actor, action, and IP filters. Inspect the action description and its Project or object context. When investigating missing results, check the selected range, filters, access, and the event types recorded by the operation.
### Export a filtered CSV
Use **Download first 1K records** to download the first 1,000 matching records with the selected filters. The export includes timestamp in UTC, user, action type, description, Project ID, and IP where recorded. It is separate from the currently displayed page size.
For a review requiring more than the exported population, narrow the date range and retain the filter and range context with each file. Confirm coverage of the review period rather than treating a 1,000-row file as a complete workspace history. Handle the download as workspace information under your organization's access and retention requirements.
### Interpret the records
Application events, infrastructure logs, approval history, and execution trajectories have distinct collection and retention scopes. The operational Log Analytics retention described in the Trust Center is not an Audit Log retention promise.
Captured execution records show the supported instrumentation and reported capture state. Interpret recorded actions alongside that capture state, and use the artifact's approval history to identify approved Policy versions and reviewer decisions.
The CSV export contains the matching application events described above. Review required event coverage, retention, delivery, storage, and integration needs as part of your security review, including any requirements for continuous delivery, immutable storage, or data-read records.
## Related task pages
{% related-card-grid title="Inspect the owning record" %}
- [Admin Console](/docs/admin-console)
- [Approval History and Reviewer Activity](/docs/governance/approval-history-and-reviewer-activity)
- [Trialground](/docs/concepts/trialground)
- [Security Review and Assurance](/docs/security/assurance)
{% /related-card-grid %}
---
id: security.data-handling
title: Data handling
summary: Trace what enters Teammately, what AI services process, and how workflow choices affect storage, processing locations, retention, and deletion.
kind: reference
product_area: security
status: stable
updated: 2026-11-11
canonical: /docs/security/data-handling
---
# Data handling
## Definition
Customer content includes more than uploaded Cases. Project instructions, Reference Materials, expert contributions, files, candidate responses, and captured tool activity can all contain business information. Synthetic experiments process and retain the selected context, constructed inputs, and resulting activity according to the chosen workflow.
## Fields, states, or lifecycle rules
| Workflow | Information processed | What to review |
| --- | --- | --- |
| Project setup and expert work | Entered instructions, materials, comments, answers, and supported recordings | Which information participants supply and who can access it |
| Case synthesis | Coverage requirements, supplied context, generation instructions, and generated Cases or materials | Whether every source is suitable for the intended processing |
| Managed Trialground execution | Selected Case inputs, materials, saved Harness, outputs, and captured trajectory events | Tools, granted credentials, model calls, returned data, and retained records |
| External execution and output submission | Benchmark Cases and criteria held in Teammately, plus submitted responses, failures, and any included trajectories | What remains in your environment and what the submission transmits |
| Connected integrations | Data selected for the specific connection and operation | Provider, access scope, processing location, and applicable terms |
### Follow the information through the workflow
For a synthetic experiment, permitted Project context and fictional requirements feed Case and material generation. The selected Cases, materials and saved Harness then enter managed execution, or your own execution environment. Submitted responses and captured activity enter assessment and the resulting Run records.
| Information | Where it enters or remains | Selection and access responsibility |
| --- | --- | --- |
| Project Agent Brief and coverage definitions | Context for construction and other selected AI workflows | Review the Brief and relevant Dimensions, Topics, Scenarios, and source descriptions |
| Sample Cases and source material | Generation context when the selected operation includes them | Remove unapproved samples before preparation; new synthesis and source exclusion are different choices |
| Inputs, generated materials and candidate code | Benchmark assets and the selected execution | Project permissions and candidate selection govern use |
| Tool requests and returned content | The Harness's configured services and supported captured activity | Endpoints, resource permissions, credential grants, and capture configuration |
| Responses and recorded execution | Run, Results, and supported execution records | Review content before external submission and apply the intended access scope |
| AI-service state | The model feature requested by the owning workflow | Provider terms, feature configuration, processing location, and retention scope |
Private ingestion and inference provide private transport for agreed paths. Review the submitted information and each service's processing location alongside that network configuration. See [Network Controls and Isolation](/docs/security/network-isolation).
### Core services and processing locations
The [Trust Center](https://teammately.ai/trust) lists Microsoft Azure for application hosting, databases, storage, backups, and AI processing; Auth0 for authentication; and Cloudflare for proxied traffic and delivery. Core Azure hosting is in Central US. Model services and traffic delivery have their own processing geographies, described below and in the Trust Center.
Teammately's platform AI processing uses Azure OpenAI or Microsoft Foundry. Global model deployments may process prompts and responses across supported geographies. Customer-selected Harness endpoints and integrations have their own processing scope.
### Model training and service storage
Teammately does not use customer workspace data to train shared generative models without the customer's express written agreement. This commitment is separate from processing content for a requested service or retaining operational records. Customer-requested weights training is a separate workflow with its own selected dataset and provider.
Model-service storage follows the selected API feature and workflow. Managed execution retains outputs and captured trajectory events outside the sandbox, where they remain available after execution ends under the applicable retention arrangements. Review [Microsoft's AI data-processing terms](https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy) alongside the features and integrations you select.
### Retention, deletion, and recovery
Your requirements should distinguish active application records, files, model-service data, logs, and backups. The applicable agreement and selected workflows determine the retention and deletion commitments. Storage soft deletion and backup recovery periods describe recovery protection. Permanent-erasure timelines apply to the agreed records and stores in the deletion scope.
The [Privacy Policy](https://teammately.ai/privacy) and Trust Center describe privacy-request handling. Customer-controlled workspace content is handled on the customer's documented instructions, with identity and authority checks where appropriate.
| Record category | Retention or deletion scope to establish |
| --- | --- |
| Active Project and Benchmark records | The selected objects, permitted use, historical versions, and applicable agreement |
| Files, generated artifacts and supported recordings | Originals, derivatives, versions, and references held by the workflow |
| Model-service records | Uploaded content and stored state requested by the selected feature |
| Application and audit logs | The relevant event collection and its retention; infrastructure-log retention is separate |
| Backups and recovery copies | Recovery windows, expiration, and the handling of previously deleted content after restoration |
Use the privacy-request process to identify the authorized objects and stores in a deletion request. Record the agreed completion criteria and treatment of historical or recovery copies. Review active records, retained execution data, and recovery copies as distinct parts of that scope.
### Recovery objectives
Teammately's published controls describe database backup and restoration procedures, storage versioning, and soft-deletion protections. Review those protections alongside the required recovery point and recovery time, then define the applicable objectives for the agreed deployment and workflows.
## Related task pages
{% related-card-grid title="Choose the intended data flow" %}
- [Synthetic Evaluation](/docs/security/synthetic-evaluation)
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Connect Model Outputs](/docs/integrations/connect-model-outputs)
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
{% /related-card-grid %}
---
id: security.network-isolation
title: Network access and IP restrictions
summary: Use workspace IP restrictions and private connectivity, and distinguish network access from synthetic inputs, execution sandboxes, and processing locations.
kind: reference
product_area: security
status: stable
updated: 2026-11-11
canonical: /docs/security/network-isolation
---
# Network access and IP restrictions
## Definition
Teammately provides workspace IP restrictions and private connectivity for agreed deployment paths. Synthetic inputs, managed execution, network access, and processing location address different requirements.
## Fields, states, or lifecycle rules
| Boundary | What establishes it | What to inspect |
| --- | --- | --- |
| Production records stay outside an experiment | Fictional inputs and materials, permitted generation context, and test-only data sources | Project Agent Brief, facets, samples, tools, credentials, and outputs |
| Candidate execution runs in a managed environment | Trialground's saved Harness and runtime configuration | Candidate identity, dependencies, state handling, and captured activity |
| Workspace access comes from allowed addresses | Enabled IP Address Control with a nonempty allowlist | The addresses seen at the application's authenticated workspace boundary |
| Selected traffic takes a private route | Private connectivity arranged for the agreed ingestion or inference path | Direction, endpoints, network ownership, and the selected service |
| Content is processed in an agreed geography | Hosting, model-deployment type, edge delivery, and integration scope | Each processing step and provider, including model calls |
### Configure workspace IP restrictions
1. Open **Admin → Security → IP Address Control** with permission to manage workspace security.
2. Enter the intended IP addresses or CIDR ranges under **Allowed IPs / CIDRs**. Include the addresses your organization actually uses to reach the application.
3. Enable the restriction and save the configuration. An enabled setting needs a nonempty allowlist to restrict access.
4. Have an authorized administrator confirm access from the intended network and the restriction from outside it, using the normal sign-in and application flow.
This setting governs authenticated workspace application access. Project permissions govern authorized actions, while Harness configuration governs outgoing connections. Access-policy lookup failures and denied addresses are different results; use the reported state when diagnosing access.
### Arrange private ingestion and inference
Open **Admin → Security → Private connectivity**. Private connectivity is available as part of your Teammately deployment, including AWS PrivateLink, Azure Private Link, or equivalent connectivity appropriate for the environment.
Review both directions separately:
- **Your network to Teammately:** private access for sending data, including Fieldloop logs.
- **Teammately to your model endpoint:** private access for model requests and responses.
The setup is arranged for your environment through the contact destination on that page. Identify the endpoints, data categories, direction, DNS and routing requirements, and responsible network owners. Review the route and processing geography for each selected service and integration.
### Keep evaluation data separate
Managed Harnesses can make network requests to their configured services. Use endpoints and credentials whose allowed resources are test-only when the experiment must exclude production records. Keep fictional data in the supporting services as well as the initial Case.
A private model connection protects its transport path. Retrieval permissions and data sources determine which records tools can return; the experiment's inputs, tools, and credential grants define its information boundary. Read [Evaluate Without Uploading Production Cases](/docs/security/synthetic-evaluation) for the complete construction and execution workflow.
## Related task pages
{% related-card-grid title="Configure the relevant boundary" %}
- [Admin Console](/docs/admin-console)
- [Access Control](/docs/security/access-control)
- [Evaluate Without Uploading Production Cases](/docs/security/synthetic-evaluation)
- [Data Handling and AI Processing](/docs/security/data-handling)
{% /related-card-grid %}
---
id: security.synthetic-evaluation
title: Evaluate Without Uploading Production Cases
summary: Use synthetic Cases, fictional materials, and a test-only Harness environment to evaluate behavior while keeping actual production records outside the experiment.
kind: task
product_area: security
status: stable
updated: 2026-11-09
canonical: /docs/security/synthetic-evaluation
---
# Evaluate Without Uploading Production Cases
Uploading production Cases is optional. You can construct the situations your agent must handle from behavioral requirements, expert judgment, and coverage design, then evaluate them in Trialground. Actual customer conversations, transactions, and confidential documents can remain outside that experiment.
Lemon Weave provides the synthesis capability: it constructs candidate Cases, controlled alternatives, and supporting materials from the context and coverage you supply. Trialground runs saved Harness Versions in managed sandboxes and assesses recorded behavior against your benchmark criteria.
## Prerequisites
- A Project whose Project Agent Brief and coverage definitions contain only information permitted for this experiment.
- A selected Benchmark, its Project Input Schema, and permission to manage coverage and candidate membership.
- A named behavior to examine and the fictional facts or materials needed to examine it.
- A Harness with test-only endpoints and credential grants when actual production data must remain outside execution.
## Inspect the generation context
Describe the task, relevant rules, exceptions, and conditions using information approved for this workflow. Use fictional names, identifiers, documents, and tool results. Match the Project Input Schema and required material formats so the Case has the structure your candidate expects.
If a real incident motivates the Case, describe the behavioral distinction without copying its private records. Generated content can reproduce sensitive facts supplied in instructions or reference material. Review inputs and materials before admitting them to the benchmark.
Generation uses Project context as well as the text you enter for one request. Inspect the **Project Agent Brief**, the selected Dimensions, Topics and Scenarios, their descriptions and source context, and any selected sample Cases. Descriptions derived from an actual incident can contain private details even when no production Case is selected.
The fictional-only boundary applies to this whole input set. A request to use fictional names is a generation instruction; it is separate from excluding sensitive source information before processing.
## Prepare new Cases through Shots
1. Open the intended Benchmark's **Coverage Design → Canvas**, then choose **Shots**.
2. Start a **New Shot** and select the relevant conditions. Choose **Add** and set **Cases to add** to the amount needed for this experiment.
3. Enter the fictional situation and construction requirements in **Additional instructions**. Leave **Sample cases** empty, or retain only samples approved for this workflow. Remove any sample that should not enter generation.
4. Choose **Stage shot**. Inspect the staged request and preview. Use **Update staged shot** if you change it before preparation.
5. Choose **Prepare staged shots** after the preview is ready. This addition path requests newly synthesized candidates. Preparation through **Rules** can reuse existing Case Pool candidates, so source behavior should be read from the operation you actually choose.
6. Inspect the prepared Cases and required materials in the Canvas preview. Review their source relationships and eligibility, select the acceptable candidates, and approve them through the preview's review controls.
7. Confirm the current editable Benchmark membership. Create the intended Snapshot and Benchmark Version before using that fixed population and its criteria for evaluation.
Shots can be configured with sample Cases. Synthesis-only creates new candidates using the Project Agent Brief, selected facet context, and any supplied samples. Review each of these inputs for the experiment's information requirements.
## Keep the execution separate
Configure the Harness to use test-only services, fictional state, and credentials scoped to test resources. Review retrieval sources, environment configuration, callbacks, tool responses, and outputs so the whole experiment follows the intended information boundary.
Trialground manages the execution environment; the Harness defines model calls, tools, and task behavior. A job can process several Cases in one process and workspace. Configure per-Case state and reset behavior when independence is required. The selected endpoints, credential grants, and network configuration define the execution's connectivity.
Use [Network Controls and Isolation](/docs/security/network-isolation) to review workspace access and private ingestion or inference separately from the candidate's data access.
## Choose where execution runs
| Execution choice | What stays separate | What Teammately receives |
| --- | --- | --- |
| Managed Trialground execution with synthetic Cases and test-only tools | Actual production records and systems can remain outside the experiment | Supplied context, Cases, materials, the saved Harness, responses, and captured activity |
| Execution in your own environment with output submission | Your application execution and unsubmitted internal records remain there | Benchmark Cases and criteria already in Teammately, submitted responses or failures, and any included trajectories or telemetry |
External execution keeps the candidate runtime in your environment. Teammately receives the outputs and trajectories you submit for assessment. Review those submissions for the information they contain.
## Review what the assessment establishes
Experts should check realism, valid conditions, expected behavior, and the distinctions each Case exercises. Freeze the intended Case population and criteria in a Benchmark Version, then inspect responses and recorded steps.
Synthetic evaluations assess behavior in the constructed situations. Use representative production observations to understand event frequency, and assess actual integrations and production behavior within the customer's approved environment.
## Object and state changes
Staging a Shot records the proposed request in the current editing session. Preparation records an execution and produces candidates for review. Approval changes the editable Benchmark population through its owning action. Saved Snapshots and historical Runs retain their recorded populations and candidate identities.
## Success criteria
- Generation context, selected samples, and supporting materials meet the experiment's information boundary.
- Prepared additions are newly synthesized, and the accepted Cases demonstrate the intended distinction.
- The selected Harness uses only the endpoints, identities, and resources appropriate for the experiment.
- The resulting Benchmark Version, candidate Version, outputs, and captured activity are identifiable.
## Common failure modes
- Starting a synthesis-only request with sensitive Project Brief or facet content still in scope.
- Leaving an actual customer Case attached as a sample.
- Confusing a Rules preparation that reuses Cases with a new Shot addition.
- Connecting a fictional Case to a production retrieval source or credential.
- Treating preparation completion as approval or assuming execution teardown deletes retained outputs.
{% example-demo title="An invented refund history" %}
A team wants an agent to recognize that a partial refund has already been issued. It defines the rule with an approved fictional policy, creates an invented order with a $120 payment and a prior $40 refund, and supplies a test-only refund service. Variants change the earlier operation's status or repeat its request identifier. Reviewers check the eligible remainder, uncertain completion, and duplicate-action handling without supplying a customer's actual order or payment record.
{% /example-demo %}
## Related workflows
{% related-card-grid title="Construct and assess the experiment" %}
- [Synthesize Cases](/docs/coverage-engineering/synthesize-cases)
- [Review Prepared Cases](/docs/coverage-design/case-review)
- [Harnesses](/docs/assets/harnesses)
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Connect Model Outputs](/docs/integrations/connect-model-outputs)
- [Data Handling and AI Processing](/docs/security/data-handling)
{% /related-card-grid %}
## Related reference pages
{% related-card-grid title="Understand the experiment's boundaries" %}
- [Network Controls and Isolation](/docs/security/network-isolation)
- [Project Agent Brief](/docs/agent-setup/project-context)
- [Project Input Schema](/docs/project-settings/input-schema)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}
## Related troubleshooting pages
{% related-card-grid title="Investigate an unsuitable candidate" %}
- [Synthetic Cases That Feel Unrealistic](/docs/troubleshooting/unrealistic-synthetic-cases)
- [Unclear Cases](/docs/troubleshooting/unclear-cases)
{% /related-card-grid %}
---
id: governance.reproducibility
title: States, versions, and reproducibility
summary: Preserve enough source context to explain and repeat correctness decisions.
kind: reference
product_area: governance
status: stable
updated: 2026-11-11
canonical: /docs/governance/reproducibility
---
# States, versions, and reproducibility
## Definition
Reproducibility means preserving enough exact identity and observable configuration to explain what was evaluated and to repeat the supported execution path. It does not mean every future execution will produce an identical stochastic output. It means a reader can distinguish changes in candidate, Case population, evaluator, Trial count, and runtime metadata instead of attributing every result difference to the model.
## Fields, states, or lifecycle rules
- Preserve Project, Benchmark, Benchmark Version, Dataset Snapshot, and Run identity.
- For managed execution, preserve the exact saved Harness Version rather than an editable draft or display label. For imported results, preserve the external candidate context and submitted responses or recorded failures.
- Preserve admitted Case, Policy, and Rubric version boundaries through the Benchmark Version.
- Record Run and Trial identities, planned Trial count, evaluator set, and recorded execution settings. Retain the managed machine allocation or the external execution context supplied with the import, as applicable.
- Retain run metadata and measured telemetry when captured; missing values remain unknown.
- Record completeness, incomplete Cases, and terminal state beside scores.
- Use canonical evaluation receipts for Lemon Goal Session candidate claims.
- Use the recorded outputs, captured trajectory events, and execution settings to explain what the experiment observed.
## Related objects
Dataset Snapshots preserve the evaluation set. Benchmark Evaluations preserves candidate, Run, settings, results, and available telemetry. Results interpret candidates inside compatible snapshot boundaries. Harness improvement adds Goal, candidate, and canonical receipt identity when evaluation drives code or Harness changes.
{% example-demo title="Diagnosing a score change" %}
Two Runs use the same Harness Version but report different pass rates. The operator confirms that one Run used Benchmark Version 6 and the other used Version 7, which added source-conflict Cases and a revised grounding Rubric. The version and completeness record explains the movement. The team avoids filing a candidate regression until it compares Runs inside the same snapshot boundary.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Benchmark Versioning](/docs/governance/benchmark-versioning)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Benchmark Evaluations](/docs/benchmark-evaluations)
- [Complete your first alignment loop](/docs/quickstart)
- [Choose your starting point](/docs)
{% /related-card-grid %}