Your Production Logs Tell You What Happened, Not What Must Work

How Enterprise AI Gets Better · Part 4 of 6: Discovering coverage

Suppose you want a knowledge assistant to answer from customer agreements. Today it handles general policy questions and routes agreements to account managers.

Its logs faithfully represent that choice. You will not find the assistant misinterpreting those agreements: it has never been allowed to answer from them. Sampling another month of routine policy traffic leaves the proposed expansion almost untested.

The product’s boundary is already written into its traffic. To expand it, discover the agreement conditions the next version must handle, then construct cases that exercise them.

Production traces expose actual sequences, failures, and unexpected interactions; LangChain describes their role in agent improvement.¹ Coverage discovery adds requirements and deliberate exploration to that experience.

Find the behavior your traffic leaves out

Product restrictions are only one source of missing evidence. Users may stop attempting tasks that failed before. A person may repair an answer before it reaches a customer. A final action may occur in another system, outside the trace.

Check the record’s unit—message, attempted task, or completed task—and whether it retains escalations, abandoned sessions, human corrections, and later outcomes. Identify workflows that never entered the path.

Benchmark composition addresses sampling and scoring a defined workload. Coverage discovery establishes what the evaluation must exercise.

Four sources of missing behavior deserve attention:

Choose combinations that could change the decision. More metadata combinations alone do not make coverage useful.

Give the investigation a stopping point tied to the next permission. For agreement answers, first cover when to apply an exception, when to reject its applicability, and when to seek clarification. Add combinations where those decisions interact, such as an expired agreement and an unavailable replacement. Record the consequential gaps still open. The team can then decide which scope to enable while directing further construction toward a named uncertainty.

Combine three sources of cases

Observed experience, authorized requirements, and deliberate exploration contribute different things.

A new product can begin with requirements; a mature product may begin with an incident. CheckList’s behavioral-testing methodology offers a precedent for organizing capabilities and test types.² Decide which distinction the next case should exercise.

Missing rules can masquerade as coverage

In an internal Teammately trial, we asked a model what variety several policies required for training material. One fixture said to follow standard refund rules and make exceptions when appropriate, without supplying those rules or exception triggers.

The refined questions exposed a conflict. One judgment recognized the missing rules. Another treated materially different responses to the same facts as valid alternatives.

An absent rule leaves the outcome unsettled. Treating that gap as permission for multiple answers would turn missing requirements into extra generation demand. Before using the judgments in an allocation, we needed to establish what the policy actually permitted.

The investigation and actual response excerpts retain the exact questions, outputs, and remaining error. A coverage team must resolve the missing requirement before assigning approved answers to new cases.

Turn the incident into a case family

Suppose the knowledge assistant cites an old internal guide instead of a current customer agreement. A reviewer gives the response a thumbs-down.

Save the interaction and identify the concern: an outdated guide, an inapplicable agreement, missing context, or a valid cautious answer the reviewer disliked. The evidence determines which work follows.

Perhaps the incident reveals a rule: customer-specific terms override a general guide only within their authorized scope. That rule creates a family of useful cases:

Changing the name adds an example; changing validity can change the required behavior. Keep the incident for regression and use the family to exercise that distinction.

If the scope or precedence is disputed, use the standards and authority framework to settle that question before assigning an expected answer.

Give construction a clear brief

A coverage map becomes actionable when it states what the next cases must establish. For the agreement example, a compact construction brief could read:

Coverage need: Determine when a customer agreement changes the answer.

Relevant conditions: Validity period, service scope, source availability, and authorized precedence.

Required contrast: Apply a current, relevant exception while declining to apply an expired or unrelated one.

Construction work: Build cases for the settled conditions with the necessary documents available.

Expert clarification: Resolve precedence when two applicable sources conflict.

Missing-evidence behavior: Ask or route when the governing agreement cannot be verified.

The brief separates a missing case from a missing decision. A generator can help with the first. More generated examples cannot authorize the second.

Use the construction brief for one rule and its changing conditions. A practitioner checks the distinction, a case builder prepares evidence and starting state, and an engineer verifies execution and outcome checks.

In Teammately’s Coverage Engineering, record the agreement boundary and select validity, service scope, availability, and precedence conditions. Those selected combinations direct Lemon Weave to construct the cases and supporting material. An unsettled precedence rule returns to expert inquiry. The credit walkthrough connects those handoffs.

Validate the cases as well as the answers

Give each generated case a reason to exist: its changed condition, why it matters, and evidence supporting the expected behavior.

To test expiration, hold the other terms fixed. A changed service clause would confound the contrast.

An agreement-citation case needs obtainable evidence, or an explicit missing-evidence purpose. An action case needs starting records, tool behavior, and outcome verification; a single-response case may need only a document and question.

Build the smallest environment that preserves the distinction. Review construction patterns, sensitive boundaries, and uncertain labels against the standard. Add simulation when it changes what the agent must perceive, decide, or do.

For the agreement family, construction review might produce three decisions:

Preserve each disposition’s reason so later changes do not erase the distinction.

Let experience refine the map

Start with the material you can legitimately use: a small authorized sample, a support incident, or a practitioner-reconstructed scenario. Minimize copied sensitive information while retaining the facts needed to interpret the case.

Before production, requirements can guide a challenge suite. Later experience should challenge it: a planned condition may prove irrelevant, while an overlooked interaction becomes urgent. Keep population estimates separate from these selected cases.

Send implementation defects to engineering, missing conditions to construction, doubtful labels to assessment review, and disputed rules to their owners. Some incidents need no change.

Keep observation, interpretation, and resulting work connected. When policy or environment changes, the team can revisit the reason for each case and preserve the basis of subsequent improvement claims.

Production reveals experience with the current product. Coverage discovery prepares the evidence for its next decision.

Continue with your question

Once the required cases are defined, read how their selection and weights change the benchmark’s conclusion.

Next: When Your Benchmark Becomes the Training Target

Browse the series and resources.

References