# Project Secrets
Generated: 2026-11-03T07:03:03.118Z
Source build: local
Canonical docs: https://teammately.ai/docs
---
id: project-settings.secrets
title: Project Secrets
summary: Manage project access to workspace secrets and grant credentials to the intended Harness without placing values in content.
kind: reference
product_area: project_settings
status: stable
updated: 2026-10-30
canonical: /docs/project-settings/secrets
---
# Project Secrets
## Definition
Open **Project Settings → Secrets** to manage the project's access to supported workspace secrets. Grant the intended credentials through the owning Harness controls instead of copying values into candidate source.
## Fields, states, or lifecycle rules
- Secret identity and access metadata are different from the secret value.
- Project access and an individual Harness grant have different scopes.
- Rotation changes the secret through its owning workspace operation.
- Archiving or removing access can affect future credential resolution.
- Existing Runs retain their recorded candidate and execution identities.
## Grant deliberately
Identify the service the candidate needs and the exact Harness that should use it. Review project and Harness access rather than assuming that workspace membership grants every candidate the same credentials.
Keep values out of the Project Agent Brief, Cases, Reference Materials, copied agent tasks, Run labels and metadata. A model or provider name is descriptive configuration, not a credential grant.
## Review changes
Use the current secret and grant controls for creation, rotation, access and archive operations. Read reported failures before retrying a logical change. A display row or masked value does not establish that a candidate can resolve the secret during execution.
When debugging, inspect the Run's configuration and authorized grant metadata. Do not paste secret values into a conversation to prove that a credential exists.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Project Settings](/docs/project-settings)
- [Harnesses](/docs/assets/harnesses)
- [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings)
{% /related-card-grid %}
---
id: project-settings.overview
title: Project Settings
summary: Maintain project understanding, Lemon instructions, Regime, input schema, members, and secret grants in the current settings surface.
kind: concept
product_area: project_settings
status: stable
updated: 2026-10-30
canonical: /docs/project-settings
---
# Project Settings
Project Settings maintains the current configuration shared by a project's Benchmark work. Open the relevant tab for the object you intend to change.
## Definition
The settings surface contains **General**, **Project Context**, **Reference Materials**, **Lemon instructions**, **Regime**, **Input Schema**, **Project Members** and **Secrets**.
| Setting | Owns | Important distinction |
| --- | --- | --- |
| General | Project name and Project Memo | Memo is human-facing text, not agent context |
| Project Context | Project Agent Brief | Stable project understanding |
| Reference Materials | Sources and indexed content | Source refresh does not rewrite the brief |
| Lemon instructions | Configured activity guidance | Project override and automatic history |
| Regime | Framework, points, gates and guidance | Settings do not approve individual artifacts |
| Input Schema | Canonical input and materials | Benchmark planning inherits this contract |
| Project Members | User and group access | Membership is separate from task assignment |
| Secrets | Project secret access and grants | Values are not Run metadata or source code |
## Change boundaries
A project setting can affect future work across multiple Benchmarks. Accepted tasks and Runs retain the inputs, settings and instructions they captured. Saving current settings does not rewrite historical results.
Changes record their actor, scope, time and before/after content automatically where the owning setting provides history. Users do not create or publish a separate settings version to make an edit take effect.
## Operating sequence
Inspect the current value and its scope. Make the intended edit, save through the owning control and read its authoritative result. Resolve a concurrent-change response without discarding another operator's work.
Use Input Schema to change required Case materials. Use Regime to configure point interpretation. Use Lemon instructions for activity guidance and the Project Agent Brief for stable product meaning.
## Related work
Benchmark Datasets owns selected membership and Snapshots. Coverage Design owns input requirements and adoption decisions. Expert Contributions owns a request, its accepted plan and attributable judgments.
Run details preserve recorded execution metadata. They do not create a current Run Metadata settings tab.
## Related reference pages
{% related-card-grid title="Related reference pages" %}
- [General Project Settings](/docs/project-settings/general)
- [Project Context](/docs/agent-setup/project-context)
- [Reference Materials](/docs/agent-setup/reference-materials)
- [Lemon Instructions](/docs/project-settings/lemon-instructions)
- [Regime Settings](/docs/project-settings/regime)
- [Project Input Schema](/docs/project-settings/input-schema)
- [Project Members](/docs/project-settings/project-members)
- [Project Secrets](/docs/project-settings/secrets)
{% /related-card-grid %}
---
id: assets.harnesses
title: Harnesses
summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
kind: reference
product_area: assets
status: stable
updated: 2026-10-30
canonical: /docs/assets/harnesses
---
# Harnesses
## Definition
A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation.
A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry.
## Fields, states, or lifecycle rules
- A newly created Harness begins with one blank editable Draft.
- Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded.
- The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts.
- Draft validation errors block publication. Warnings remain visible for operator review.
- **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also requests use of that Version for the selected Benchmark; confirm its **In use** state.
- A previously saved Version can be selected with **Use this Version** for the current benchmark.
- Existing Runs keep their referenced Harness version even after later Draft or Saved changes.
- Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets.
- Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata.
- Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft.
- Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`.
- Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation results.
## Draft, saved version, and activation
Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began.
The Harness editor owns **Publish** and **Use this Version**. Those actions select a saved Version for the current Benchmark. The evaluation launch chooses saved Versions, machine and Trials; it does not provide the retired activation-settings control. Every admitted Run keeps its exact saved Version, so a later activation change does not rewrite earlier results.
## Runtime, debug, and output mapping
Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run.
Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness.
Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable.
{% example-demo title="Example: exact candidate identity" %}
An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work.
{% /example-demo %}
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Start a Lemon Goal Session](/docs/improve/start-improvement-session)
- [Project Input Schema](/docs/project-settings/input-schema)
{% /related-card-grid %}
---
id: benchmark-evaluations.execution-settings
title: Evaluation Execution Settings
summary: Understand saved candidate selection, machine allocation, Trial counts, and the conditions preserved by a Run.
kind: reference
product_area: benchmark_evaluations
status: stable
updated: 2026-10-30
canonical: /docs/benchmark-evaluations/execution-settings
---
# Evaluation Execution Settings
## Definition
Execution settings determine which saved candidate runs, how much compute it receives and how many Trials execute against the fixed Benchmark Version. Review these choices in the launch flow.
## Fields, states, or lifecycle rules
- A managed candidate is an exact saved Harness Version. The editable Draft is not its execution identity.
- The launch selects an execution machine and freezes its allocation for that Run.
- A Trial executes the full planned Case population once. Repeated Trials are independent executions.
- The service accepts a whole-number Trial count from 1 through 50.
- Accepted Runs preserve their original configuration after project defaults or candidate activation change.
## Candidate selection
Publish a coherent Harness Draft and choose its exact saved Version for use. Activation makes that Version available in the relevant benchmark workflow. Archiving or deactivating a Harness does not rewrite historical Runs.
An imported response set has no executable Harness bundle. It can be inspected and compared in Results but cannot be activated for managed execution.
## Machines and defaults
The launch form requires an explicit machine selection. Available machines describe execution contracts independently of billing. CPU and memory requirements declared by the Harness still apply.
API and Lemon Goal callers can resolve compute from their explicit selection, then project, workspace and product defaults. Benchmark-level compute defaults do not control new work. Existing admitted Runs keep their recorded allocation.
## Comparing settings
Candidate behavior is comparable only when the Benchmark Version, evaluation configuration, machine and planned Trial population support the comparison. A label or creation timestamp does not establish matching conditions.
Run evaluation reuses matching work. Run again requests an independent repeat under recorded conditions. The reference Run used for score is selected before outcomes are available; extra repeats do not become a larger hidden sample.
Use Run history to inspect other executions. Read repeated metrics with their aggregation rule and the recorded Trial count rather than today's launch controls.
## Related task pages
{% related-card-grid title="Related task pages" %}
- [Run a Benchmark Evaluation](/docs/benchmark-evaluations/run-evaluation)
- [Evaluation Results](/docs/benchmark-evaluations/results)
- [Harnesses](/docs/assets/harnesses)
{% /related-card-grid %}