{"query":"Run a Benchmark Evaluation","corpusVersion":"local","generatedAt":"2026-11-18T14:23:14.721Z","results":[{"blockId":"benchmark-evaluations.run#run-a-benchmark-evaluation","pageId":"benchmark-evaluations.run","title":"Run a Benchmark Evaluation","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#run-a-benchmark-evaluation","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"run-a-benchmark-evaluation","kind":"task","productArea":"benchmark_evaluations","score":2875.7342134241794,"reasons":["search_match","title_match","display_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"# Run a Benchmark Evaluation\n\nUse a managed evaluation to execute a saved Harness Version against every Case in the selected Benchmark Version."},{"blockId":"benchmark-evaluations.run#prerequisites","pageId":"benchmark-evaluations.run","title":"Prerequisites","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#prerequisites","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"prerequisites","kind":"task","productArea":"benchmark_evaluations","score":1676.1678658901683,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Prerequisites\n\n- A Benchmark Version with the intended Cases and eligible evaluation rules.\n- A saved Harness Version with its runtime and required secret grants.\n- Permission to run evaluations and an available execution machine."},{"blockId":"benchmark-evaluations.run#steps","pageId":"benchmark-evaluations.run","title":"Steps","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#steps","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"steps","kind":"task","productArea":"benchmark_evaluations","score":1639.0340037147887,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Steps\n\n1. Open the Benchmark's **Benchmark Evaluations → Runs**.\n2. Choose **Run evaluation** and select the saved Harness Versions. Confirm each Version rather than relying on the Harness name.\n3. Select the execution machine. Its CPU and memory allocation must meet the Harness requirements.\n4. Keep one Trial for a single execution, or select **Repeat Trials** and enter the total Trials per Run, including the first.\n5. Check the displayed Case executions and confirm the launch.\n6. Follow output preparation and rubric evaluation separately. Responses can finish before their judgments.\n7. Open Results after checking the planned population and completeness.\n\nThe service accepts 1–50 Trials per selected candidate. The launch form checks a positive whole number; the service enforces its upper bound. This is a per-launch choice rather than a request to change a benchmark-wide sampling profile."},{"blockId":"benchmark-evaluations.run#object-and-state-changes","pageId":"benchmark-evaluations.run","title":"Object and state changes","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#object-and-state-changes","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"object-and-state-changes","kind":"task","productArea":"benchmark_evaluations","score":1397.7466200483543,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Object and state changes\n\nA new admission records one Run per selected candidate, its Trials, exact Harness and Benchmark Versions, execution allocation and evaluation settings. A matching admission returns the existing Run. Recorded responses, judgments and telemetry accumulate independently.\n\nThe Run automatically selected as **Used for score** is chosen before results are known. Later repeats remain in Run history and do not silently replace it."},{"blockId":"benchmark-evaluations.run#reuse-and-deliberate-repeats","pageId":"benchmark-evaluations.run","title":"Reuse and deliberate repeats","pageTitle":"Run a Benchmark Evaluation","url":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","humanUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation#reuse-and-deliberate-repeats","markdownUrl":"https://teammately.ai/docs/benchmark-evaluations/run-evaluation.md","sectionId":"reuse-and-deliberate-repeats","kind":"task","productArea":"benchmark_evaluations","score":1314.2975442273782,"reasons":["search_match","page_title_match","term_match","prefix_or_fuzzy_match"],"markdown":"## Reuse and deliberate repeats\n\nRun evaluation reopens a Run with matching immutable conditions when one already exists, including a queued, running, completed or failed Run. It does not start extra Trials merely because the action was selected again.\n\nUse **Run again** when another independent execution is intended. A repeat keeps the referenced conditions. Changed conditions must be handled explicitly rather than silently joining a different population."}]}