{"query":"Run a Benchmark Evaluation","corpusVersion":"local","generatedAt":"2026-09-13T04:38:04.441Z","results":[{"blockId":"benchmark-evaluations.run#run-a-benchmark-evaluation","pageId":"benchmark-evaluations.run","title":"Run a Benchmark Evaluation","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#run-a-benchmark-evaluation","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"run-a-benchmark-evaluation","kind":"task","productArea":"benchmark_evaluations","score":3149.8098811920263,"reasons":["search_match","title_match","display_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"# Run a Benchmark Evaluation\n\nLaunch a managed evaluation when the immutable Benchmark Version, governed evaluators, and candidate runtimes are ready."},{"blockId":"benchmark-evaluations.run#steps","pageId":"benchmark-evaluations.run","title":"Steps","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#steps","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"steps","kind":"task","productArea":"benchmark_evaluations","score":999.4744758257316,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Steps\n\n1. Open **Benchmark Evaluations** for the intended Benchmark Version.\n2. Open Evaluation Settings if you need to adjust the machine configuration.\n3. Start a Run and select one or more offered Harness Versions. Confirm the exact version labels rather than relying on Harness names alone.\n4. Choose the number of Runs for each Harness. Counts may differ; review the total execution volume.\n5. Supply any requested Run Metadata. Keep credentials out of descriptive fields.\n6. Launch. Each selected Harness creates its own Run Group containing the requested independent Runs, including when the count is one.\n7. Follow output and evaluation progress. Distinguish queued, running, complete, failed, cancelled, and incomplete work rather than inferring completion from partial scores.\n8. Inspect List, Dashboard, Arena, or Compare only after checking which attempts and Cases are evaluable."},{"blockId":"benchmark-evaluations.run#related-reference-pages","pageId":"benchmark-evaluations.run","title":"Related reference pages","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#related-reference-pages","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"related-reference-pages","kind":"task","productArea":"benchmark_evaluations","score":981.7978133846229,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Related reference pages\n\n{% related-card-grid title=\"Related reference pages\" %}\n- [Evaluation Execution Settings](/docs/benchmark-evaluations/execution-settings)\n- [Harnesses](/docs/assets/harnesses)\n- [Run Metadata](/docs/benchmark-evaluations/run-metadata)\n{% /related-card-grid %}"},{"blockId":"benchmark-evaluations.run#evidence-created","pageId":"benchmark-evaluations.run","title":"Evidence created","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#evidence-created","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"evidence-created","kind":"task","productArea":"benchmark_evaluations","score":979.0538006823468,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Evidence created\n\nThe launch creates Run Groups and Runs bound to exact Harness and Benchmark Versions. Per-Case outputs and evaluator outcomes accrue separately, so output completion can precede evaluation completion. Provider telemetry can include tokens, cost, and latency when captured; absence of telemetry is not zero usage.\n\nDashboard aggregates compatible observed Runs across launches. Choose average score, passed at least once, or passed every time where supported. Each Run retains its own outputs and status; inspect the group and individual Runs when work is incomplete.\n\n> No Draft execution\n>\n> A managed benchmark Run does not evaluate the mutable Harness Draft. Save the candidate and select its exact saved Version when launching.\n\n{% example-demo title=\"Example: two candidates, three attempts\" %}\nHarness Versions 6 and 9 are active with `n=3`. One launch creates two Run Groups and six independent Runs against the same Benchmark Version. If one attempt fails preparation, the group reports incomplete evidence instead of silently treating the remaining two as the configured cohort.\n{% /example-demo %}"},{"blockId":"benchmark-evaluations.run#prerequisites","pageId":"benchmark-evaluations.run","title":"Prerequisites","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#prerequisites","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"prerequisites","kind":"task","productArea":"benchmark_evaluations","score":965.9351022060308,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Prerequisites\n\n- A Benchmark Version backed by the intended Dataset Snapshot.\n- Approved eligible Policies and Rubrics.\n- At least one saved project Harness Version.\n- Prepared Harness runtime and required secret grants.\n- A chosen number of Runs for each selected Harness."},{"blockId":"benchmark-evaluations.run#object-and-state-changes","pageId":"benchmark-evaluations.run","title":"Object and state changes","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#object-and-state-changes","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"object-and-state-changes","kind":"task","productArea":"benchmark_evaluations","score":962.4383902562398,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Object and state changes\n\nLaunching creates one Run Group per Harness and one or more independent Runs. Outputs, evaluator outcomes, progress, metadata, and telemetry accrue to those records. A later launch creates new evidence and does not overwrite the cohort."},{"blockId":"benchmark-evaluations.run#success-criteria","pageId":"benchmark-evaluations.run","title":"Success criteria","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#success-criteria","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"success-criteria","kind":"task","productArea":"benchmark_evaluations","score":954.9402372282658,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Success criteria\n\n- Exact Harness and Benchmark Versions are recorded.\n- Each launch group contains the number of Runs requested for that Harness.\n- Output and evaluation progress reach an interpretable terminal state.\n- Incomplete or failed attempts remain visible."},{"blockId":"benchmark-evaluations.run#source-confidence","pageId":"benchmark-evaluations.run","title":"Source confidence","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#source-confidence","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"source-confidence","kind":"task","productArea":"benchmark_evaluations","score":954.2628803688264,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Source confidence\n\nCode-backed: the current Run modal, Runs workspace, and Run Group route define selection, group creation, repeated attempts, progress, and evidence identity."}]}