Should Your Benchmark Represent Reality—or Deliberately Distort It?

How Enterprise AI Gets Better · Part 6 of 6: Composition and claims

The same agent can score 97.1% on one benchmark and 78.25% on another without changing its behavior.

At a launch review, the difference matters: are you estimating normal workload success, or testing the conditions behind a new permission?

For a worked example, assume a support workload with four mutually exclusive groups and the following shares and pass rates:

A pass means satisfying the applicable task requirements. For a restricted request, that can mean correctly declining or escalating.

Weight the pass rates by workload frequency:

0.80 × 99% + 0.15 × 94% + 0.045 × 80% + 0.005 × 40% = 97.1%

Now give the four groups equal weight:

(99% + 94% + 80% + 40%) ÷ 4 = 78.25%

The first averages the assumed workload. The second gives each group equal influence. The agent did not get worse. The question changed.

Figure 1. Same assumed behavior, different weights.

Download the figure as SVG.

Rare restricted requests barely affect the workload average. Their 40% pass rate may still decide whether the agent receives a permission. That decision needs the group result.

Give the benchmark a specific job

A useful evaluation can serve three different purposes:

Define a workload estimate’s population: attempted tasks for which users, workflow, and period? Include or explicitly exclude human-routed requests. Define success as the intended reply or completed outcome.

Required-condition testing deliberately exercises rare exceptions, missing evidence, and permission boundaries. Choose its mix for those conditions.

A diagnostic pair can vary only whether an approval has expired. It tests that behavioral distinction; a suitable workload sample estimates how often the condition occurs.

The purposes can share cases while retaining their different interpretations.

Rare conditions need deliberate sampling

In the example workload, restricted-action requests account for 0.5% of traffic. Under independent random sampling, 200 requests have about a 36.7% chance of containing none of them:

(1 − 0.005)^200 ≈ 0.367

Even a representative sample can leave that requirement unexamined. Deliberately collect cases from the rare group.

Figure 2. Equal weighting gives each group 25% of the benchmark’s weight. Choose the allocation for your decision.

For a permission requiring authorization, report unauthorized actions separately. Routine successes must not cancel those violations in the readiness decision.

Match the evidence standard and system safeguards to the permission and its consequences. Successful tests cover the exercised conditions; future reliability remains an inference with limits.

Preserve the work while changing its mix

Oversampling an uncommon valid task preserves the workflow. Inventing impossible tool behavior or requiring an answer from inaccessible evidence changes it.

Coverage discovery establishes why each case belongs and whether its expectation is valid. Composition then decides how much influence those cases should have in this evaluation.

Preserve sources, applicability, and dates for document tasks; preserve state, permissions, tool effects, and outcome checks for actions. Increase difficulty through relevant conditions. Include a hard case for the requirement it exercises.

Reweight only estimates that can support it

Extra examples can support a weighted population estimate when their within-group samples represent the workload groups and the workload proportions are known.

The hardest known exception failures form a selected challenge group. Their pass rate cannot represent all exceptions, even after weighting. Untested groups and unverified outcomes remain gaps.

Frequency weights estimate performance across a population. Priority or loss weights express the importance assigned to outcomes. Label the interpretation of the aggregate.

Giving unauthorized actions more influence expresses a business priority. It does not predict that those requests occur more often. The priority score can help compare candidates, while the workload estimate answers how they perform across the defined traffic. Keep both meanings visible.

These four groups form a partition. For overlapping labels, define aggregation and denominators without double-counting requests.

Report breadth and precision together

A benchmark may exercise many conditions but provide little evidence for each. Another may estimate a narrow behavior precisely while missing important conditions altogether.

A hundred variations of one agreement may provide less breadth than cases covering different applicability boundaries. Account for their shared lineage before treating them as independent samples.

Repeated trials measure consistency on selected tasks. τ-bench separates repeated-trial reliability from a single successful attempt.¹

Report distinct tasks and trials per task. Document sampling and dependence before attaching population confidence statements. Even 100 independent successes do not establish a zero failure rate; the calculation note explains the assumptions and bound.

For a consequential condition, show violations, evidence counts, unresolved cases, scope, and the decision rule.

Make the denominator visible. Count applicable, assessable criteria as pass or fail. Report inapplicable criteria, insufficient evidence, and unresolved expectations separately.

Missing information can itself be the test: an agent that invents an unavailable document can fail. Missing outcome evidence creates a different problem. If neither the action record nor authoritative state establishes completion, keep the outcome unresolved; passing the reply’s other criteria does not establish task success.

Suppose 100 tasks yield 90 confirmed successes, five failures, and five unresolved outcomes. Excluding the unresolved tasks produces 94.7% among assessed outcomes. Across all 100 tasks, confirmed success is 90%; even if every unresolved task succeeded, it would be 95%. That 90–95% range describes the missing outcomes in this batch, not a population confidence interval. Recovering the evidence may be the next useful investment.

Keep criterion and task denominators separate too. Nine passing criteria out of ten do not make a task successful when the failed criterion is required authorization. State which applicable requirements must pass together.

Use a scorecard that preserves the questions

A concise report can provide three views:

These views can lead to different actions at the same meeting. The workload result can support using the assistant for routine requests, the required-condition result can keep a particular action under human approval, and the diagnostic result can fund the next engineering experiment. Preserving all three prevents a single average from settling decisions it was never designed to answer.

In the worked example, the workload figure is 97.1%. The required-condition view exposes the 40% restricted-request result. A diagnostic view could investigate whether authorization evidence changes those failures. Each supplies information the others omit.

Use the benchmark decision scorecard to state the claim, denominator, case-selection method, and evidence counts before reporting the aggregate.

Keep missing customer exceptions and unsettled expectations visible in the readiness report.

Report completion, quality, and handoffs with clear denominators. Routing difficult work to people can raise success among completed tasks while reducing automation. Separate human-assisted and autonomous outcomes.

Report assessment exposure alongside composition. Changing the mix leaves the history of development feedback intact.

Write the claim before building the suite

Start the next benchmark review with one sentence:

We will use this evaluation to decide whether ______, for ______, under ______ conditions.

Selecting an assistant for routine requests needs a workload estimate. Permitting autonomous account changes needs authorization-boundary evidence. Explaining expired approvals needs controlled contrasts.

Specify the evaluation unit, case selection, score interpretation, and required evidence. Report each purpose in its corresponding view.

Bring the scorecard to the next review. The series guide connects it to standards, coverage, diagnosis, and assessment.

A benchmark should represent the question you need answered. Write that question before choosing its proportions.

Continue with your question

If the benchmark has repeatedly guided candidate changes, read what that exposure means for an improvement claim.

Read the series from the beginning: Who Should Define What “Good” Means for an AI Agent?

Browse the series and resources.

References

Assumptions and calculations · Sampling calculation note