The Model Is Capable. Why Does Enterprise AI Still Fail?

How Enterprise AI Gets Better · Part 2 of 6: Failure diagnosis for leaders

In a notebook, the model reads the records, understands the task, and produces an answer an experienced colleague would accept.

Inside the application, the same model gives the wrong answer. The team tries a longer prompt, a more expensive model, and another retrieval strategy. A few examples improve. Others get worse.

For the leader deciding what to fund or launch, each change spends time and money on a different explanation. A successful demonstration establishes a capability; a dependable workflow must deliver it under the conditions of the work.

Before funding the next change, identify the explanation it will test.

The demonstration supplied more than a model

Suppose an internal research assistant accurately summarizes an account from records an engineer supplies.

The deployed system must identify the account, retrieve current records, preserve them through the interaction, and distinguish a commitment from a colleague’s suggestion. The demonstration’s engineer may already have solved those problems.

Anthropic’s context-engineering guidance treats instructions, tools, external information, and conversation state as parts of the agent’s available information.¹ The demonstration supplied that working environment along with the model.

The question to ask is concrete: which useful condition was present in the demonstration and absent in the deployed workflow?

An applicable exception, a clean transaction record, or a clarified requirement points to a different investment.

Better retrieval can leave the outcome unchanged

Sourcegraph’s published coding-agent comparison makes this distinction tangible. In a 288-task study, both groups had access to identical repository revisions. Structured retrieval increased file-level F1 from 0.091 to 0.240, yet the overall task-completion reward stayed essentially flat. A subsequent review classified about 74% of the tasks as solvable using grep alone.²

Many tasks in that corpus left little room for better retrieval to improve completion. The investment question is which work needs the proposed capability, and whether its benefit reaches the outcome the business needs.

One wrong promise, six different investigations

Consider an order-management assistant. A customer asks whether a replacement can arrive by Friday. The assistant confirms that it can. The order does not qualify for that delivery date.

“Hallucinated a delivery promise” describes the symptom. It leaves the repair open.

Several hypotheses may contribute. Compare complete alternatives: a clearer interface may let the existing model succeed, while a stronger model may compensate for confusing information. Reliability and operating cost determine which investment is useful.

Inspect the assessment before assigning the repair. Anthropic documents ambiguous tasks and graders that reject valid solutions among the sources of apparent agent failure.³

Ask two questions in the postmortem

Ask why this run produced the wrong result, then why the team missed the condition or failed to keep it fixed. The first directs the repair. The second directs coverage, ownership, and regression work.

For the delivery assistant, the immediate defect could be a stale lookup after a destination change. The development-process defect could be that every test kept the destination fixed.

Repair the lookup, then test destination changes. More static delivery questions would leave the missed transition unexamined.

Engineering can own stale-state handling. A process owner can settle when a delivery estimate becomes a permissible promise. The evaluation team can establish whether the relevant condition is represented and judged correctly.

Require evidence for the proposed investment

A trace shows what occurred. The next experiment tests the explanation.

If missing information is the hypothesis, check whether supplying valid evidence changes behavior across related cases. If execution is the hypothesis, examine the state transition that went wrong. If the model is the hypothesis, compare candidates under documented information, tool access, and operating constraints.

Require a hypothesis, a discriminating check, and a statement of what success would establish. Part 3 develops the engineering experiments.

In an internal Teammately trial, a refund policy received an applicability score with zero reported confidence. Inspection found that our question mixed conditional refund eligibility with an unconditional duty to explain. We narrowed it to eligibility and revised the response categories. With the policy and model unchanged, the revised question elicited the expected classification: eligibility depended on interacting conditions. The recorded investigation shows that correction and a remaining error. The next work was question design.

A useful failure brief can fit on one page:

Workflow: Confirm delivery dates for standard replacement orders.

Observed failure: A Friday commitment followed a change of destination without a fresh availability check.

Consequence: The customer received a commitment the order could not meet.

Current explanation: The workflow retained an availability result whose destination no longer applied.

Next evidence: Check the stale-state hypothesis across changed-destination cases and confirm that unchanged orders still work.

Decision owner: Engineering owns state handling; the workflow owner confirms the evidence required to make a commitment.

Deployment question: Does the repair support the intended commitment authority, or does this condition still need human review?

Attach the run evidence to the failure brief. The technical lead prepares the hypothesis; the workflow owner confirms the permitted behavior; the investment owner decides which investigation to fund.

Keep three kinds of progress distinct

An investigation can improve the implementation, correct the assessment, or change the product requirement. The launch meeting needs to know which occurred.

If the business decides the agent should give estimates and route confirmed commitments to a person, the team has narrowed its authority. That may be a sensible launch decision. Record the new scope and its handoff cost.

If an evaluator was wrong, correct it and rerun the baseline and candidate under the same method. If the implementation improved, preserve the requirement and show the behavioral change.

One rising score can conceal whether the team solved the task, repaired its measurement, or selected a different task.

Fund the explanation you can test

Use the failure brief to commission the experiment that best distinguishes the explanations.

Compare successful alternatives on task outcomes, latency, cost, operational complexity, and human handoffs. A system that escalates every difficult request may have changed the automation tradeoff even if its conditional success rate rises. Carry those outcomes into the operating-value comparison before translating a benchmark gain into a business return.

An investigation ends with a more specific investment: repair stale-state handling, settle commitment authority, or test a model’s recurring limitation. Require that specificity before approving the change.

Continue with your question

If the investigation reveals disagreement about what the agent should have done, start with standards and authority.

Next: What Should You Fix: the Model, the Harness, the Context, or the Specification?

Browse the series and resources.

References