Teammately Docs
Docs menu

reference

Evaluation Execution Settings

Configure machines and choose saved Harness Versions and Run counts at launch.

Evaluation Execution Settings

Definition

Evaluation Settings contains machine settings for managed execution. Harness selection and repetition are choices made when launching an evaluation.

Fields, states, or lifecycle rules

  • Choose an exact saved project Harness Version when launching. Benchmark activation is not required.
  • Choose a whole number of Runs from 1 through 50 for each selected Harness. The default is one, and different Harnesses may have different counts.
  • Each managed Harness launch creates one Run Group containing the requested Runs, including a one-Run group.
  • Existing Runs keep their immutable configuration and evidence. Launching more Runs creates another group rather than rewriting previous membership.

Reliability choices

Dashboard, Compare, and Arena aggregate compatible observed Runs for each saved Harness Version across launch groups. Average score weights individual Runs equally. Passed at least once and passed every time summarize observed binary case outcomes where the evaluation framework supports them.

Run counts can differ. The product displays the counts and coverage and warns about unequal evidence without requiring another launch. Missing or infrastructure-failed observations are not numerical successes or failures.

Before launching

Check the exact saved Version, machine selection, and requested execution volume. Runtime preparation and required access still apply. Imported external outputs remain a separate flow because they have no executable Harness Version.

Related task pages

Reference / AssetsHarnessesBuild versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.Task / Benchmark EvaluationsRun a Benchmark EvaluationLaunch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups.

Source confidence

Code-backed: Evaluation Settings defines machine configuration; the shared Run modal defines saved-Version selection and per-launch Run counts.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Benchmark Evaluations Benchmark Evaluations Run and inspect exact Harness Versions against an immutable Benchmark Version through Dashboard, List, Arena, and Compare. Benchmark Evaluations Run a Benchmark Evaluation Launch exact active Harness Versions against an immutable Benchmark Version as standard or repeated Run Groups. Assets Harnesses Build versioned Python Harness bundles, validate and debug Drafts, publish exact Versions, and activate them for benchmark evaluations.
AI context