# Trialground Generated: 2026-09-13T04:36:40.150Z Source build: local Canonical docs: https://teammately.ai/docs --- id: concepts.trialground title: Trialground summary: Evaluate exact Harness and Benchmark Versions in a managed proving ground with inspectable responses and Rubric evidence. kind: concept product_area: trialground status: stable updated: 2026-09-07 canonical: /docs/concepts/trialground --- # Trialground Trialground is the evaluation capability inside Teammately correctness infrastructure. It gives AI engineers a managed proving ground for running exact candidate implementations against exact Benchmark Versions while preserving the response, Rubric outcomes, settings, and metadata needed to explain the result. ## Definition A candidate is represented by a saved Harness version rather than an informal description of a prompt or code state. A benchmark version provides the cases and governed correctness boundary. Benchmark Evaluations binds those inputs into Runs, computes rubric-level evidence, and exposes results through Dashboard, List, Compare, Arena, and run-detail views. Trialground names the capability; **Harnesses** and **Benchmark Evaluations** are the current UI surfaces. The distinction matters when giving instructions: users edit and save a Harness under Assets, then select its saved version for a benchmark evaluation. ## Decision checkpoint | Question | Inspect | Required boundary | | --- | --- | --- | | What code or prompt was evaluated? | Harness and saved version | Draft editor state is not the candidate identity | | What behavior space was tested? | Benchmark version and dataset snapshot | Dataset membership is fixed for interpretation | | Which standards judged each case? | Applicable policies and rubrics | Suggested or unrelated standards are excluded | | Why did a candidate pass or fail? | Case response and applicable Rubric outcomes | Aggregate movement is traced to Case-level evidence | | Can two candidates be compared? | Compare or Arena | Both candidates expose compatible benchmark and settings boundaries | ## Managed evaluation Harness Assets provide a multi-file editing surface, Draft and Saved states, coding-agent assistance, secret references, and benchmark activation. Evaluation setup selects the relevant Harness version and preserves run metadata. Managed execution can scale case work while keeping every result attached to the benchmark and candidate identities. The current Benchmark Evaluations API does not expose execution trajectories, spans, private reasoning, or tool traces. The visible **Traces / Spans** segment is therefore a capability fence, not inspectable Run evidence. Improve's narrated trajectories are separate session chronology and must not be described as evaluation execution traces. Rubric outcomes remain the governed evidence within the evaluation contract. ## Reading evidence Start with Dashboard to understand overall progress and distribution, then use List for Run and Case-level results. Compare is a symmetric matrix whose columns are saved Harness Versions and whose rows can group Cases, all results, Policies, Rubrics, Dimensions, Topics, Groups, or Patterns. Arena computes pair evidence from admitted results; it is not a subjective preference interview. Run detail is the place to resolve execution state, output mapping, response evidence, and failures. When a result is surprising, check versions and configuration before changing the candidate. A different benchmark snapshot, Harness version, output mapping, evaluation setting, or run-metadata value can explain movement that otherwise looks like a behavior change. {% example-demo title="Retrieval candidate comparison" %} Two saved Harness versions use different source-selection strategies. Both run against the same benchmark version containing current-source, stale-source, and missing-source cases. Compare shows that one candidate improves current-source grounding but regresses when no controlling source exists. The team opens the failed cases and rubric results before deciding whether the candidate belongs in the next Improvement Session. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Start an Improvement Session](/docs/improve/start-improvement-session) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Harnesses](/docs/assets/harnesses) - [Benchmarks](/docs/object-model/benchmarks) - [Run metadata](/docs/benchmark-evaluations/run-metadata) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines Trialground as the public capability. Harness and Benchmark Evaluation pages provide code-backed behavior and state details. --- id: assets.harnesses title: Harnesses summary: Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations. kind: reference product_area: assets status: stable updated: 2026-08-22 canonical: /docs/assets/harnesses --- # Harnesses ## Definition A Harness is a project-level, versioned Python bundle used to produce behavior for benchmark cases. It implements `generate_result(request)` at its configured entrypoint and returns the result expected by the case contract. The workspace provides a multi-file editor, one editable Draft, immutable saved Versions, Coding Agent proposals, secret grants, debug runs, runtime preparation, export, and benchmark activation. A Harness can represent prompt logic, retrieval, tools, orchestration, or other candidate behavior supported by the managed runtime. It is the executable candidate boundary, not a general model registry entry. ## Fields, states, or lifecycle rules - A newly created Harness begins with one blank editable Draft. - Draft source and runtime changes autosave after a short delay. Concurrent edits use a Draft revision and can surface a conflict that must be reloaded. - The Draft contains source files, an entrypoint, Python version, dependency specifications, sandbox profile, and target-platform/runtime facts. - Draft validation errors block publication. Warnings remain visible for operator review. - **Publish** creates an immutable saved Version from the exact Draft bundle and runtime configuration. Publication also uses that Version for the currently selected benchmark. - A previously saved Version can be selected with **Use this Version** for the current benchmark. - Existing Runs keep their referenced Harness version even after later Draft or Saved changes. - Archived Harnesses remain visible but cannot be edited, debugged, activated, or granted new secrets. - Credentials belong in workspace secrets and are explicitly granted to the Harness. Secret values do not belong in source, case content, or Run Metadata. - Coding Agent proposals are staged as file diffs. Accept or reject the proposed hunks before they modify the canonical Draft. - Runtime delivery can include the rendered case view, static `case_material_refs`, and an optional supported `world_instance_ref`. - Debug runs execute only a saved Version against a `harness_contract.v1` request. They are authoring checks, not benchmark evaluation evidence. ## Draft, saved version, and activation Use Draft for active editing. Publish when a candidate is coherent enough to identify, compare, or evaluate. If a benchmark needs a different saved Version, select it explicitly. Do not describe a Run as evaluating “the current Harness” because the Draft and benchmark activation may have changed after the Run began. The Evaluations execution settings show project Harnesses with saved Versions and control which are active for future managed Runs. Activation is benchmark-scoped. Every admitted Run still records the exact saved Harness Version, so later activation changes do not rewrite existing evidence. ## Runtime, debug, and output mapping Publishing preserves bundle and dependency hashes and can prepare the immutable runtime image. A failed or missing runtime can be prepared or retried from the saved Version. Use Debug to send a single contract-shaped request before spending a benchmark Run. Output mapping is a separate boundary. A Harness return must match the benchmark's declared result contract. For imported reference outputs, Benchmark Evaluations maps uploaded rows to immutable benchmark cases. Neither debug success nor a syntactically valid upload proves rubric correctness. Improvement Sessions can generate or receive candidate Harness versions. Those candidates become meaningful in a session only when their identity and canonical evaluation receipt are observable. {% example-demo title="Example: exact candidate identity" %} An engineer edits retrieval filtering in a Harness Draft and publishes Version 12 for the selected benchmark. A managed Run records that exact Version. The engineer then continues editing the Draft toward Version 13. The existing Run still refers to Version 12, so its case responses and rubric results remain attributable even though the editor shows newer work. {% /example-demo %} ## Source confidence Code-backed: the active Harness list and editor establish creation, archive state, Draft autosave and conflicts, validation, publication, activation, secrets, debug, Coding Agent proposals, and runtime preparation. The runnable adapter supports the execution boundary without making its internal API public. ## Related task pages {% related-card-grid title="Related task pages" %} - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Configure Project Input Schema](/docs/project-settings/input-schema) {% /related-card-grid %} --- id: benchmark-evaluations.overview title: Benchmark Evaluations summary: Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare. kind: concept product_area: benchmark_evaluations status: stable updated: 2026-09-13 canonical: /docs/benchmark-evaluations --- # Benchmark Evaluations Benchmark Evaluations is the version-scoped workspace for executing and comparing candidate systems. The active top-level tabs are **Dashboard**, **List**, **Arena**, and **Compare**. Every managed Run binds an exact saved Harness Version to the immutable Benchmark Version shown in the route. > Evaluation boundary > > Interpret evidence inside its recorded Benchmark Version, Harness Version, Run or Run Group, evaluator set, and metadata. Run counts belong to launches. Additional launches add evidence without rewriting earlier Runs. ## Surfaces and objects Dashboard summarizes progress, leaderboards, rank progression across Runs, and available resource telemetry. List is segmented into **Runs**, **Evaluation results**, and **Traces / Spans**. The results segment summarizes Case outcomes and Policy or Rubric failures. Arena compares candidate pairs across governed metrics. Compare is a symmetric matrix of Harness Versions across selected evidence rows. A Run Group can collect one standard attempt or repeated attempts. A Run records one candidate execution and its per-Case progress. Evaluation results record the admitted Policy and Rubric outcomes. Costs, tokens, and latency are telemetry only when the provider or execution path captured them. > Traces / Spans capability fence > > The List navigation exposes Traces / Spans, but the current benchmark API does not expose evaluation execution traces. Do not claim that trajectories, spans, private reasoning, or tool traces can be inspected from Benchmark Evaluations today. ## Decision checkpoint | Need | Open | Evidence to preserve | | --- | --- | --- | | Configure and launch managed Runs | Evaluation Settings and New evaluation run | Machine, saved Harness Versions, and per-Harness Run counts | | Start candidate execution | Run modal | Exact Harness and Benchmark Versions | | Inspect status and output summaries | List → Runs or Evaluation results | Run Group, attempt, Case counts, incomplete state | | Compare candidate pairs | Arena | Metric family, pair count, only-A, only-B, shared failures | | Compare many candidates by governed rows | Compare | Harness columns and chosen Case or facet row mode | | Admit external reference outputs | Output mapping | Case mapping, attempt assignment, insert/update report | ## Rankings and repeated sampling Dashboard aggregates compatible observed Runs for each saved Harness Version across launches. Average score weights Runs equally. Supported binary views report passed at least once or passed every time over the observed case outcomes. Counts and missing evidence are shown; unequal counts do not prevent comparison. Historical group metrics retain their recorded meanings. Ranking is a routing signal. A candidate can lead overall while failing required Policy or high-impact Rubric evidence. Use Arena or Compare to locate the disagreement and List to confirm completeness before starting Improve work. ## External outputs Uploaded or API-supplied reference outputs create output-only Runs that can be scored and inspected in List. They are not saved Harness Versions and therefore cannot be optimized in Improve or selected as Harness columns in Compare or Arena. {% example-demo title="Example: repeated evaluation without evidence drift" %} A team launches three Runs of Harness Version 8 and one Run of Version 11 against the same Benchmark Version. Both appear with their evidence counts. A later launch of Version 11 adds two Runs to its aggregate evidence without changing either launch group. The team can inspect individual Runs before deciding whether more evidence is useful. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Configure evaluation execution](/docs/benchmark-evaluations/execution-settings) - [Run a benchmark evaluation](/docs/benchmark-evaluations/run-evaluation) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Use Arena and rankings](/docs/benchmark-evaluations/arena-and-rankings) - [Compare Harness Versions](/docs/benchmark-evaluations/compare) - [Map external outputs](/docs/benchmark-evaluations/output-mapping) {% /related-card-grid %} ## Source confidence Code-backed: the active version-scoped workspace, settings, Run modal, List segments, Dashboard, Arena, and Compare routes define the current evaluation model and capability fences. --- id: concepts.coevolve title: Coevolve summary: Explore multiple evidence-backed candidate directions while keeping goals, benchmark receipts, trajectories, and the current frontier connected. kind: concept product_area: coevolve status: stable updated: 2026-08-22 canonical: /docs/concepts/coevolve --- # Coevolve Coevolve is the improvement capability inside Teammately correctness infrastructure. It lets agents and engineers pursue multiple candidate directions from benchmark evidence, evaluate those candidates through the same canonical path, and continue from stronger branches without losing the goal, chronology, or evidence behind the current frontier. ## Definition Coevolve is realized through **Improve** and durable **Improvement Sessions**. A session starts from selected benchmark evidence and a target Harness. A Goal Contract turns a free-form objective into pinned target identities, measurement bindings, constraints, unresolved items, and permitted intervention boundaries. Once confirmed, that contract governs future work without rewriting earlier evidence. Sessions can use Work or Evolve behavior. Work supports bounded implementation and review. Evolve supports parallel candidate exploration across epochs and retains a frontier based on current evaluation evidence. Candidate Harness versions become meaningful only when canonical Runs return observable results. ## Decision checkpoint | Situation | Improvement action | Evidence requirement | | --- | --- | --- | | One known candidate change needs implementation | Start a Work session | Confirmed Goal Contract and pinned target evidence | | Several hypotheses should compete | Start an Evolve session | Explicit authorization, measurement bindings, and comparable evaluation path | | An external coding worker will implement | Prepare a scoped worker package | Returned Harness version or evaluation request before claiming observable progress | | A candidate looks stronger | Inspect the current frontier | Canonical evaluation receipts support the retained position | | Exploration exposes missing correctness or coverage | Return the observation upstream | Identify the policy, rubric, case, or coverage artifact that must change | ## Evidence-backed branching Candidate exploration is not a sequence of undocumented edits. Each proposal should state the hypothesis and its relationship to the Goal Contract. Evaluation receipts bind candidate identity to benchmark identity and result. Narrated trajectories can explain the work performed, while chronology records durable transitions, agent activity, pauses, resumptions, and terminal state. The current frontier is not simply the newest candidate. It represents the candidates retained by the session's evidence and goal constraints. A candidate can improve one slice and regress another; the frontier and comparison views keep that tradeoff visible. ## Bridge between experts and engineers Improvement can reveal that the candidate is not the only incomplete part of the system. A missing coverage tuple, unclear rubric, contradictory policy, or insufficient case material should become an upstream contribution opportunity. This is how Coevolve connects coding agents to domain experts: engineering work is guided by benchmark evidence, and newly discovered correctness questions return to focused expert work. External workers remain bounded. Teammately may prepare a package for Codex, Claude Code, or another worker, but it records only the work returned through the defined contract. It does not infer private activity or fabricate a working state. {% example-demo title="Parallel grounding hypotheses" %} An Evolve session starts from failures involving conflicting policy documents. One candidate changes retrieval filtering, another changes source ranking, and a third changes answer construction. Each saved Harness version is evaluated against the pinned benchmark. The frontier retains the candidates supported by grounding and uncertainty rubrics, while a newly observed source-authority ambiguity becomes an Expert Contribution opportunity. {% /example-demo %} ## Related workflows {% related-card-grid title="Related workflows" %} - [Start an Improvement Session](/docs/improve/start-improvement-session) - [Inspect evaluation results](/docs/benchmark-evaluations/inspect-results) - [Request an Expert Contribution](/docs/expert-contributions/request-contribution) {% /related-card-grid %} ## Related reference pages {% related-card-grid title="Related reference pages" %} - [Improve](/docs/improve) - [Harnesses](/docs/assets/harnesses) - [Product boundaries](/docs/introduction/product-boundaries) {% /related-card-grid %} ## Source confidence Doctrine-backed: this page defines Coevolve as the public capability. The Improve pages provide code-backed session, contract, candidate, and frontier behavior.