Teammately Docs
Docs menu

task

Benchmark Dataset Cases

Inspect benchmark Case membership, coverage traces, references, and scoped bulk actions.

Benchmark Dataset Cases

Prerequisites

  • A selected benchmark and permission to inspect or manage its current dataset.
  • Project Cases that conform to the intended Input Schema.

The Cases tab is the benchmark-scoped view of the current editable case set. It shows Case content and membership together with coverage trace, output or reference mapping, and evaluator relationships.

Select one or more rows to request an Expert Contribution, create another benchmark from the selection, remove the Cases from the current benchmark, or download them. Removal changes current membership; it does not delete the reusable Case from project Assets or mutate an existing Snapshot.

Review before snapshotting

  1. Confirm each Case still conforms to Project Input Schema and has the intended materials.
  2. Inspect Coverage Facet assignments and source or contributor provenance.
  3. Check policy and rubric application, including whether eligible evaluator links are approved.
  4. Resolve missing or ambiguous output/reference mapping when the workflow requires reference outputs.
  5. Use Representation to check whether the set supports the intended claim.

After changing membership

Open Representation and confirm that the change affected the intended facet or evaluator population. Removing redundant Cases can improve balance even when total Case count falls. Adding many near-duplicates can increase count without adding meaningful coverage.

If a selected Case needs content correction, edit it through the owning Case workflow and review every future benchmark that selects it. Existing Snapshots stay unchanged. If the Case reveals an unclear standard, request an Expert Contribution before compensating with more examples.

Worked example

Example: scoped bulk action

An operator selects five Cases tied to an unresolved exception and requests one Expert Contribution. The Cases remain in the current set while the expert works. After the controlling Rubric is clarified, the team reviews membership and creates a new Snapshot with the approved evaluator links.

Object and state changes

Selected-row removal changes current benchmark membership; creating another benchmark creates a separate benchmark; requesting a Contribution creates scoped expert work. Downloads and inspection are read-only. No action here mutates an existing Snapshot.

Success criteria

  • Current membership, Case identity, coverage trace, and evaluator relationships are understood.
  • Any bulk action affects only the intended selected Cases.
  • A new Snapshot is created when changed membership must become evaluation evidence.

Common failure modes

  • Treating removal from the benchmark as project-level Case deletion.
  • Assuming editable membership changed an old Benchmark Version.
  • Selecting Cases by visible text while ignoring their durable IDs.

Related reference pages

Concept / Benchmark DatasetsBenchmark DatasetsSelect benchmark Cases, inspect representation, and freeze immutable Snapshots for reproducible evidence.Reference / AssetsCasesUnderstand canonical project cases, their input and materials, and how they become members of benchmark datasets.Reference / Project SettingsProject Input SchemaDefine the canonical input architecture, case-material fields, and accepted artifact formats for project cases.

Related troubleshooting pages

Troubleshooting / Recovery LibraryCase upload troubleshootingRepair uploads with rejected rows, missing inputs, incorrect column mapping, or Cases that arrive without usable context.Troubleshooting / Recovery LibraryUnclear CasesRepair Cases that informed reviewers cannot interpret consistently from the supplied input, context, and output identity.Troubleshooting / Recovery LibraryUnbalanced CoverageFix benchmarks that overrepresent easy or common cases while missing important behavior.

Source confidence

Code-backed: the active Cases route defines the benchmark membership table, coverage trace, selected-row operations, downloads, and output/reference presentation.

Found something unclear?

Report outdated, unsupported, or confusing docs so we can fix the source page.

Report a docs issue

Continue learning

Related docs

Benchmark Datasets Benchmark Datasets Select benchmark Cases, inspect representation, and freeze immutable Snapshots for reproducible evidence. Benchmark Datasets Dataset Representation Analyze how distinct benchmark Cases are distributed across facets, evaluator rules, and provenance. Assets Cases Understand canonical project cases, their input and materials, and how they become members of benchmark datasets. Expert Contributions Request an Expert Contribution Create a focused benchmark contribution with an accountable expert, clear objectives, selected cases, attachments, and appropriate task components.
AI context