Who Should Define What “Good” Means for an AI Agent?

How Enterprise AI Gets Better · Part 1 of 6: Standards and authority

Suppose an account-management agent offers a customer a credit to resolve a billing complaint. The customer accepts. The conversation is polite, the arithmetic is right, and the ticket closes.

Did the agent do a good job?

Customer success says yes: it saved the relationship. Finance says no: the account does not qualify. The account manager points to an exception negotiated last quarter. Engineering asks which answer belongs in the evaluation set.

The disagreement determines whether the agent can issue credits autonomously, must request approval, or should decline this request. A prompt revision cannot settle it.

Expertise, authority, and implementation are separate responsibilities. Evaluation needs the decision they produce: a standard engineering can implement and the next reviewer can apply.

Every evaluation carries a decision

A test that rewards issuing the credit encodes one decision. A test that rewards refusing it encodes another. Calling either answer “ground truth” does not establish that the decision was authorized.

Braintrust’s Evals are the new PRD makes the case for expressing product expectations through repeatable tests. Mercor’s discussion of credible agent evaluations emphasizes judgment from people who understand the work.¹²

Who can turn an informed judgment into a standard that applies beyond this case?

A senior account manager may understand why a credit makes sense without having authority to establish company policy. A finance owner may approve the policy without knowing every operational exception. An engineer may encode both faithfully without having a mandate to resolve their disagreement.

Separate the goal, the standard, and the evidence

The goal describes the useful result. For the account agent, that might be resolving billing questions accurately while avoiding unnecessary handoffs.

The standard describes the required behavior and its scope. When is a credit allowed? Which records must be checked? Can the agent execute the adjustment, or only recommend it?

The evidence establishes whether the standard was met. A fluent reply shows communication quality. A transaction record shows that an adjustment occurred. An approval record shows whether it was authorized. Each supports a different conclusion.

Customer preferences and support costs inform the decision; the approval establishes permission. Organizational authority settles rules and tradeoffs. It cannot make an incorrect account balance or a false contract claim true. Those facts still need evidence.

A small ownership map is often enough:

Start with the engineer bringing the run evidence, a practitioner who understands the exception, and the owner authorized to approve its interpretation. One person may hold several responsibilities. Involve other owners when the rule crosses their authority or exposes a conflicting requirement.

Not every disagreement needs a new policy. Suppose two bags both meet a shopper’s budget, capacity, and delivery needs. A retail expert may prefer the lighter one; the shopper may prefer the cheaper one. Capture the tradeoff and the acceptable range. An evaluation that insists on the expert’s favorite bag would turn a preference into an obligation. The same ownership model distinguishes a permission to act from a judgment among valid alternatives.

Turn a judgment into a reusable standard

Suppose the account manager explains that the disputed credit was approved in a renewal agreement.

Recording “the agent was correct” fixes a label. Capturing why makes the judgment useful for the next account.

Where is the approval recorded? Does it cover this service period and amount? Has the credit already been issued? Does the exception authorize execution or require a further approval? What should the agent do when the record cannot be verified?

The answers can become a compact standard:

Decision: Apply an account credit.

Applies when: A current, account-specific approval covers the requested credit, and the credit has not already been issued.

Required behavior: Verify the approval and transaction state before acting. Explain the result without making additional commitments.

Missing evidence: Request the necessary information or route the request to the designated owner.

Ownership: The process owner approves the rule and scope. Practitioners supply exceptions and examples. Engineering implements and tests it.

Use the standard-record worksheet to capture the rule, its authority, and the contrasting cases that explain it.

The rubric can now ask: did the agent verify a current approval before applying the credit? Instructions, tool restrictions, cases, training material, and human review can all carry that same decision. An expired approval changes the answer without changing the standard.

Route disagreements to the decision they reveal

Begin by checking ordinary mistakes: a missed fact, an outdated rule, or the wrong account record. When defensible judgments still conflict, identify what remains unsettled.

The minority reviewer may be the only person who noticed the exception. Before voting on labels, compare the evidence behind them.

Make a practitioner’s preferred sequence mandatory when it serves an explicit requirement. Verifying approval protects a dependency; reproducing the practitioner’s phrasing may simply preserve a habit.

Keep unresolved rule judgments visible, alongside the number and relevance of cases affected. Continue assessing agreed requirements, such as checking the account or routing an unresolved approval.

Make expert review concrete

Start with the credit behavior the team wants to enable. The engineer prepares the disputed run and draft record; the practitioner checks the exception; the accountable owner settles its scope.

Show a current approval, an expired one, and one belonging to another account. Ask whether the owner endorses those different outcomes for those reasons. Reviewing the contrasts can reveal an overbroad exception that a polished policy paragraph conceals.

AI can prepare comparisons, proposed criteria, and follow-up questions. Review the interpretation against the evidence and authority before adopting it. Our coverage-judgment investigation shows how a plausible machine judgment can conceal a missing rule.

In Teammately, expert contributions turn that reasoning into proposed criteria. For the credit, the review question becomes: does this approval authorize this account, period, and amount? Rubrics and Policies retain the approved scope, precedence, and supporting context. Your organization owns approval. The credit walkthrough carries the record into coverage and evaluation.

Leave the meeting with something engineering can use

Leave the review with the standard, its required evidence, the behavior when evidence is missing, and the accountable owner.

Keep the disputed example and its contrasts beside that record. Engineering can implement and evaluate the behavior, then return with a specific question if it fails.

The standard provides continuity through the series operating model as teams evaluate and change the system.

Before closing, give a second reviewer an unfamiliar case and the record. Ask them to identify the applicable rule, decisive evidence, and permitted action. If they need the original expert to explain the exception again, the record is still missing part of the decision.

The next customer will bring different details. The standard earns its value when the next person can apply it.

Continue with your question

If your rule is settled but its exceptions are missing from the tests, read how to discover coverage beyond production logs.

Next: The Model Is Capable. Why Does Enterprise AI Still Fail?

Browse the series and resources.

References