Stop Blaming Every Agent Failure on the Model: From Coarse Error Codes to a Verifiable Failure Chain
When an agent fails to deliver, first locate the stage that demonstrably failed rather than automatically blaming the model. Only then can we decide whether to retry, repair a protocol, inspect a tool, or fix delivery.
We encountered a failure that looked like PROVIDER_UNAVAILABLE. The provider request had in fact completed with HTTP 200 and a full stream; the refusal came later from Publication validation. Switching models and raising timeouts would have repaired the wrong part of the system.
Think of parcel tracking: “dispatched” is not “delivered.” Likewise, a successful model API response does not prove that the report reached the user's screen.
A research task also crosses parsing, tools, persistence, and actual user receipt. “No answer” is a symptom, not a root-cause finding. Our diagnostic work therefore starts with recorded execution facts and their relationships, not with a model-generated explanation of what supposedly happened.
1. We Had Enough Logs, but No Reliable Relationships Between Them
Model calls, tools, Publication, databases, and frontend streams all produce logs. They often exist as unrelated fragments.
A timeline might read:
14:03:01 model request started
14:03:07 provider 200
14:03:08 candidate parsed
14:03:08 publication rejected
14:03:09 another model turn started
14:03:15 block committed
14:03:15 stream closedIt is tempting to tell a complete story: the first answer was rejected, the model read feedback, fixed it, committed the revision, and delivered it.
But three relationships are not established by that sequence:
- Did the next model request actually consume the rejection feedback?
- Was the later committed block really a revision of the rejected block?
- Did the user actually receive the committed block?
Temporal proximity is not causality. We needed both stages and identities/receipts linking legitimate effects.
Read the relationships in groups; the complete diagram remains available below.
- → Model response
- Correlation identities and receipts → User delivery [dashed relation]
- → Parse result
- → Tool execution
- → Publication handling
- → Persistence
- → User delivery
Complete relationship diagram · enlarge for detail
This sketches possible stages. Only shared Run identities and actual receipts establish connections; neighboring timestamps alone do not.
Missing classification does not make a failure minor
During the long-run investigation recorded on October 30–31, 2026, three runs contained internal request-stage failures and unconfirmed terminal settlement, but lacked identifiable failure codes and locations. Identical generic messages do not establish a shared root cause. Co-occurrence of settlement and request-stage errors does not establish their causal direction.
The continuation supervisor had required failureCode before requesting a review of an internal failure, so it missed these unclassified cases. Its revised detection recognizes unclassified internal failure and terminal-settlement failure, reads back the current batch, then pauses the next batch for review. It does not clear the original failures, reset admission, or rewrite saved text.
This repairs supervisory detection, not the underlying Runtime causes. The next investigation needs bounded exception-type and code-location evidence from the original path, not blanket retries. The independent protocol-observation issue is covered in trace-completeness case in the observability panorama.
2. First Improvement: Last Confirmed Blocker, Not Imagined Root Cause
We considered fields like rootCause or primaryFailure and rejected them.
Suppose:
Provider returns 200
→ JSON parses successfully
→ Publication validation rejects
→ No content is deliveredWe can establish that publication validation was the last confirmed barrier to complete delivery. We cannot yet establish why it rejected: malformed model content, an overly strict rule, invalid upstream data, or a contract mismatch are all possibilities.
So the diagnostic field is terminalBlocker, not rootCause:
terminalBlocker.stage = publication_validation
terminalBlocker.entityRef = original publication identity
delivery = noneThis is a conceptual excerpt, not a complete API response: the actual terminalBlocker contains a stage and object identity, rather than a bare string. It makes a narrower, defensible claim: given the verified facts, this is where the chain was definitively stopped. That is already enough to prevent irrelevant model-provider troubleshooting.
3. HTTP 200 Is Only One Successful Hop
API debugging creates strong habits: 200 means success, 4xx client error, 5xx server error. In an Agent, provider HTTP success confirms only one segment.
provider response
→ stream assembly
→ JSON parse
→ schema validation
→ Runtime orchestration
→ tool execution
→ Publication validation
→ Publication commit
→ deliveryParsing deserves its own failure identity. The current classification contract places invalid or truncated JSON under response_parse and forbids reporting it as PROVIDER_UNAVAILABLE.
One controlled live audit observed STRUCTURED_OUTPUT_INVALID_JSON. That audit did not provide an HTTP status, so it is not presented here as a measured case of failure after HTTP 200. It illustrates the distinction between parsing and transport failure, nothing stronger.
Without stage ownership, we had patterns such as:
Invalid output schema → provider unavailable
Business deadline exceeded → provider timeout
Explicit user cancellation → provider timeout
Publication rejected → provider unavailableThe provider becomes a dumping ground for unrelated errors.
Timeout attribution particularly benefited from one simple field:
timeoutOwner =
provider
| business
| user
| unknownThe layer actually initiating the abort records this fact. It is not inferred by the model or guessed from nearby timestamps. Provider timeout and a product-wide business deadline require completely different fixes.
4. Diagnosis Does Not Require Duplicating the Answer Body
Capturing every user question, model response, and raw tool result in the diagnosis store would create a larger privacy and operations problem.
We deliberately separated diagnostic facts from content. The JSON below illustrates a bounded diagnostic excerpt, not a complete API response or a real customer run:
{
"runId": "...",
"timeline": [
{
"stage": "provider_dispatch",
"status": "ok"
},
{
"stage": "response_parse",
"status": "ok"
},
{
"stage": "publication_validation",
"status": "rejected",
"reasonCodes": ["..."]
}
],
"summary": {
"terminalBlocker": {
"stage": "publication_validation",
"entityRef": { "type": "publication", "id": "example-block" }
},
"delivery": "none"
}
}The long-lived record needs allowed reason codes, stages, status, delivery facts, and explicit unknowns—not the answer text.
Raw body access may occasionally be needed during a tightly scoped incident investigation. It should not silently become permanent diagnostic storage.
Diagnosable does not mean "retain every original byte." Stable identities and bounded metadata often establish where to investigate without exposing sensitive research contents.
5. A Was Rejected and C Was Committed: That Does Not Mean A Was Repaired
Consider:
A rejected
→ Model continues
→ C committedReaders may silently convert this to "the model fixed A." But C may be an independent block, may answer an entirely different question, or may have existed before A failed.
Sequence alone proves no repair. A defensible repair chain instead resembles:
A revision 1 rejected
→ Feedback consumed by model turn 3
→ A revision 2 proposed
→ A revision 2 validated
→ A revision 2 committed
→ A revision 2 deliveredEach step has a different identity and evidentiary meaning.
Even proof that a later model turn happened does not establish that it consumed feedback. We need an explicit relation such as:
{
"actionId": "...",
"proposedByModelTurnId": "turn-2",
"consumedByModelTurnId": "turn-3"
}A stricter implementation verifies that the actual next request transcript was bound to the earlier feedback.
Without such links, a "self-repair success rate" calculated from neighboring timeline events can be a fictional number.
6. The Model Receiving Feedback Is Not the Model Understanding It
Suppose we prove that the rejection feedback appeared in the next model request. That proves the model had an opportunity to see it.
It does not prove comprehension or that the eventual answer correctly applied the feedback.
System-observable stages are:
Feedback included in request: verifiable
Later version linked as revision of A: verifiable
Later version passed validation: verifiable
Later version committed: verifiable
Later version sent: verifiable at the corresponding layer
Model genuinely understood the correction: not established by these factsThis limit is inconvenient but necessary. A diagnostic system must not become a speculative interpreter of the model's mental state.
7. Publication Commit Is Not Delivery
In a streaming Agent, a committed block is a strong signal—yet it still does not mean the user saw it.
Possible states include:
validated
→ committed
→ projected / emitted
→ client received
→ renderedNot every product measures every stage. The discipline is to claim no more than the last observed one.
If only commit is proven, say "the content was committed," not "delivered" or "seen by the user." A server-send record supports a narrower transport statement. An actual client ACK, sequence acknowledgment, or rendering receipt can support later claims.
Agents often turn "the next step probably occurred" into "the next step did occur." Diagnosis must resist precisely that shortcut.
8. Feedback and Observability Can Mislead the Repair Itself
Stage-specific records still need scrutiny: was the observer correct, and was the effect read back from its real owner?
| Observed symptom | Facts to distinguish | Actual repair locus and related case |
|---|---|---|
| Strict JSON evaluator cannot parse a score | Truncated evaluator output versus weak tested answer | The evaluator itself can fail; fixing score formatting does not improve answer quality |
| Missing Trace turn or body | No invocation versus invocation with missing capture | Langfuse capture limits and missing-body receipts; telemetry absence is not execution absence |
| Recovery still shows old text | Model failed to revise versus revision never persisted | Compare original block identity, proposal, preview, commit, and readback; not just the final page |
In the third case, the model did submit a complete revised body, the tool accepted a replaces proposal, and preview showed the revised article. A nonstandard Factory path coordinated commit without persisting the replace effect.
After correcting that path using the original Recovery identity, a deterministic regression observed the new body persisted and included in the next model request.
The original attribution—"the model cannot write a complete final answer"—was withdrawn. Correcting persistence still does not establish higher writing quality.
These historical incidents came from separate execution windows.
9. A Bounded Investigation: Verify the Step and Its Identity
The earlier incidents reduce to four questions. Was the request actually dispatched? Did the response parse? Did permitted tool and publication effects happen? Which exact committed revisions entered the delivery chain? Connect each answer to the original Run, request, and artifact identity rather than inferring links from nearby events.
A conceptual excerpt, not a complete JSON API response, can show the last confirmed blocker without claiming a root cause:
terminalBlocker.stage = publication_validation
terminalBlocker.entityRef = original publication identity
delivery = none
classificationConfidence = partialpartial does not mean “maybe successful.” It means required facts remain unavailable. Matching aggregate counts, for example, do not prove delivery of the right revisions:
Expected: A B C
Receipts: A B B
Count: 3Totals cannot replace per-block and per-revision identity. A later success also needs proof that it revised the rejected artifact, the feedback actually reached the next model request, and the corresponding delivery stage was observed. Otherwise the outcome remains unknown, not a convenient story about self-repair.
10. Diagnosis Must Not Break the Business Path
Instrumentation is often described as "side-channel," and then written like this:
await writeDiagnosticEvent()
await executeTool()Or:
const result = await executeTool()
await writeDiagnosticEvent()
return resultA slow diagnostic sink now blocks actual dispatch or return.
Our review found such risks in another lifecycle logging path: awaiting start logging could delay tool dispatch; awaiting settle logging could delay wrapper completion.
That does not prove production database lock contention occurred. It shows that "it's observability code, therefore non-blocking" is an unjustified assumption.
For each genuinely side-channel projection, test business behavior with:
diagnostics off
diagnostics on
diagnostic sink slow
diagnostic sink failedOnly then may one claim non-interference, and only for the tested scope. Proving one diagnostic projection safe does not certify every logger in the repository.
Prefer "this path did not change the business trace under these test conditions" over an unrestricted "observability is non-blocking."
11. Sometimes "Unknown" Is More Useful Than an Incorrect Error
One counterintuitive outcome of better diagnostics was more explicit unknowns:
delivery = unknown
repairOutcome = unknown
usage = unknown
cost = unknownPreviously these gaps might have been guessed as zero, false, or failed.
But no captured cost does not mean zero cost. No observed delivery receipt does not prove nothing arrived. A later successful block does not prove a prior rejection was repaired. Lacking positive provider availability evidence does not justify inventing a provider-unavailable failure.
The job is to keep known, unknown, and inferred separate.
If an apparently complete diagnostic system has almost no unknowns, it may be inventing more certainty than it measures.
12. What Diagnosis Can Establish—and What It Cannot Replace
The engineering path now distinguishes failures by stage, traces whether tool results enter subsequent model requests, and separates committed answer text from actual delivery. A rejection no longer has to be attributed generically to the model or Provider, and diagnostic capture must not alter the business outcome.
Locating the last confirmed blocker is the starting point of root-cause investigation. Cross-revision delivery identity, long-term recovery, and cost effects need evidence from their respective executions.
This diagnostic approach applies method 5, “Choosing verification and diagnosing failure”: locate the last confirmed blocker and its owner before deciding whether to repair code, add observation or retain an unknown outcome. The most visible error code is not necessarily the cause of the entire task's failure.
Conclusion: Check Three Facts Before Blaming the Model
Do not begin with “Why did the model fail?” Begin with “Which stage is proven to have failed?” For an undelivered research task, investigate in this order:
- Locate: Was the request submitted? How far did the provider, parser, tools, publication, persistence, server delivery, and client actually progress?
- Link: Did later success repair the same artifact and revision? Did the model really receive the earlier failure feedback? Adjacent timestamps are not proof of causation.
- Decide: Should we fix that component, add a necessary observation, or preserve an unknown outcome and stop unproductive retries? Diagnosis itself must not acquire the right to cancel unrelated lawful work.
This does not automatically establish root cause. It prevents a costly alternative: treating the most visible error code as the plan, repeatedly retrying a healthy component, and ignoring the point where delivery actually stopped.
An accurate account of what remains unknown is often more useful than a complete but invented failure story.
Related Mechanism: A User Rating Is Not an Error Verdict
Diagnostics establishes where failure was actually observed. A user's negative rating first says only how they evaluated one completed answer—not that a defect was verified, or that training or trading was authorized. See User Feedback as a Recorded Fact for the separate identity and readback rules.