When Every Module Can Say No: Agent Run Termination, Stage Budgets, and Safe Recovery
A failed read should stop that read, not automatically erase everything the system has already accomplished. Continuing is not permission to retry forever: authorization, cancellation, total budgets, and unknown effects may still require a stop.
Consider a teaching example. A user uploads three authorized product documents and asks an assistant to compare them and save a report. The first two are readable; the third is temporarily unavailable. The assistant should disclose the gap and deliver comparisons that stand on their own. It must not turn “unread” into “no differences,” or delete the results already saved. This illustrates a design choice, not a recorded production run.
The reading module usually knows only “I could not read this file.” It does not know whether other evidence is sufficient, another permitted route exists, or a partial result would be useful. Deciding whether one operation can proceed and deciding how the whole task should end are different responsibilities.
1. Separate a failed step from an ended task
The three-document example makes the decisions concrete:
| Decision | Who should establish it? | What it must not decide automatically |
|---|---|---|
| Was the third document read successfully? | The tool that actually reads it | That the first two documents are also invalid |
| How many searches or model requests remain? | The program that owns that resource balance | That changing stages grants a new allowance |
| Which answer revision is saved and displayable? | The existing Publication component | That the answer correctly resolves the whole question |
| Should the task continue, finalize, or stop? | The lifecycle holding original execution facts | New permission, rewritten receipts, or a substituted goal |
The lifecycle is the logic managing one research execution, called a Run, from start to settlement. It is not another supervisory Agent. It receives established facts and selects a next step that is still allowed.
Do not look only for a literal deny. Throwing an exception, returning early, or filtering a candidate answer may all turn a local failure into a global stop. Ask: Which effect does this decision prevent, and is the evidence sufficient?
2. A research note should not become a master switch
ETF Agent has used an optional list of questions called Research Needs. Think of it as a note on the analyst's desk: useful for organizing work, but neither an authorization document nor a certificate of completion.
An earlier implementation compared that list again after preparing a model request. A change could block the entire dispatch. Intended as a consistency check, it gave optional notes authority over otherwise valid work.
In the teaching example, imagine the assistant is ready to check document two. Its note changes from “check price” to “check price and validity period,” and the system cancels the whole request. The important checks are whether the request belongs to the original user, tools remain in scope, and resources remain available. A changed note alone is not enough to veto it.
Reviewer advice revealed a related issue: preparation established its identity, but a later reinterpretation could still block an admitted request. That redundant check was removed before the termination consolidation; the later work must not take credit for every earlier repair.
A Stop Hook should likewise remain advisory. It may suggest another useful investigation, but cannot independently veto a valid ending or force an extra model turn. Conversely, an empty response with neither deliverable text nor a valid call still needs a bounded, honest outcome. Removing a veto does not turn an empty response into success.
3. Permission to continue is not a fresh budget at every step
A failed read of document three may justify another existing reading method, but it must use the original permissions, balances, and deadline.
If search grants two retries, tools grant another two, and model continuation grants two more, nested attempts can turn a small failure into a long wait. Read-only operations also consume fees, concurrency slots, and upstream capacity; they are not free.
Different resources may have different owners. Search quota, model concurrency, and answer-revision capacity are not the same ledger. The problem is making two independent decisions over the same consumption, or resetting a balance when execution moves to another stage.
Exhausted search quota does not necessarily prevent reading a source already acquired. An exhausted research allowance does not necessarily consume the original reserve for final output. The next action depends on the actual remaining resource, not a generic “budget exhausted” label. See Run Budget and Constraint Governance.
4. Before adding a timeout, find out where the time went
In one historical controlled case, the page showed roughly 360 seconds elapsed: around 347 seconds of queuing and 13 seconds of execution, with execution ending normally. In another, a tool returned in approximately 0.15–0.29 seconds, yet the page showed a network error. The next model continuation had produced no answer text for roughly 30 seconds.
These are historical observations, not current performance baselines. They explain why retrying a successful tool cannot repair downstream continuation, and why extending execution timeouts may not solve queue delay.
| Clock | Starts at | What to inspect first |
|---|---|---|
| Queue time | Waiting for an execution slot | Capacity and concurrency |
| Execution time | Actual start of the stage | Model, tool, or storage behavior |
| No-progress time | Last meaningful work | Whether new usable results are appearing |
| Overall deadline | Start of the original task | Whether the whole task still has time |
A transport heartbeat proves connection activity, not research progress. Individual waits must also fit inside the original task deadline. Otherwise every stage can wait legitimately while the overall task never ends.
5. Do not replace the question to disguise non-completion
An early fixed-planner path exposed another failure. When a model did not produce a valid plan, recovery switched to generic knowledge and discarded the original entities, dates, and evidence requirements. One historical request accumulated approximately 672 seconds of planning and queuing.
The user wanted a current fact; the system returned a general explanation and marked the task complete. Similar topic, different task.
The historical repair kept an explicit planning failure instead of silently proceeding with a changed question. That is a lesson from the old architecture, not a requirement that today's native tool loop always produce a fixed plan.
For the three documents, an honest result says: “Documents one and two were compared; the third was not read, so differences across all three remain unresolved.” Replacing it with generic advice about choosing products does not complete the comparison. Degradation may reduce the claim; it must not silently replace the user's goal.
6. Let the existing lifecycle choose continuation, finalization, or stopping
We did not add a dedicated failure-deciding Agent. The existing lifecycle needs to own the actual decision to continue, recover, finalize, or settle—not merely write failed after other modules have made that decision independently.
Read the relationships in groups; the complete diagram remains available below.
- → LocalDecision
- → ContinueWork
- → EscalateFacts
- → RunLifecycle
- → BoundedRecovery
- → MainFinalization
- → DurableSettlement
- → ContinueWork
- → Publication
- → DurableSettlement
Complete relationship diagram · enlarge for detail
The branches map directly to user-visible work:
| What happened? | Possible response | Constraint to retain |
|---|---|---|
| Document three fails; two documents and other tools remain usable | Continue authorized work or deliver an explicitly scoped partial result | Do not turn missing evidence into a successful empty result |
| Temporary service failure; original recovery conditions and balance still hold | Retry within the existing bounds | Do not reset task identity, permission, or budget |
| Research cannot continue, but the original output reserve remains | Summarize acquired material and explain gaps | Failed finalization cannot return to research to claim another allowance |
| User cancellation, a hard limit, or invalid required identity | Stop new execution and settle from original facts | Do not add a model call merely for a graceful ending |
| An external effect remains unknown | Have the original execution and recovery logic reconcile it | Neither assume success nor blindly execute again |
The goal resembles Microsoft's Bulkhead pattern: limit a local failure's impact. Here it does not imply creating a separate thread pool for every module; the issue is how far one refusal may reach within an existing system.
Controlled verification of the termination consolidation supports these specific control behaviors, not live-model research quality. Acceptance should test both sides: necessary denials remain, and independent authorized work is not stopped incorrectly.
7. Recovery begins by checking what happened, not by doing it again
A connection can break after a report save. The file may not exist, or it may have been written while its acknowledgment was lost. AWS's discussion of idempotent APIs and safe retries addresses the same problem: no received response does not mean no server-side effect.
Recovery should first inspect the original task, stored results, supported checkpoints, and remaining resources. Confirmed effects are read under their original identities. Unknown effects cannot be replayed merely because someone says “try again.” A checkpoint is not a promise to recover from every failure.
Under ETF Agent's current contract, continuation within the same Run inherits recorded authorization rather than re-adjudicating that Run from today's permission table. Account, scope, content identity, cancellation, and resource checks still hold. A new Run needs its own admission. Downloads have separate access checks: research authorization is not permanent permission to download a file.
A Child follows the same separation. A closed physical model request does not prove that the Child delivered an acceptable result. After permitted continuation paths are exhausted, the Child may settle its own technical failure, but cannot rewrite the original request's usage, retry on its own authority, or erase Main's committed answer. If request completion itself is uncertain, missing text is not enough to establish failure.
A post-action checkpoint can incorrectly veto the next step. In a controlled fault injection, a tool read had already completed and left its original execution receipt. Serialization then failed before the corresponding recovery checkpoint was written. The old handling restricted the Run to finalization only, blocking an otherwise independent read. The repair did not ignore every checkpoint error: it isolated this specific pre-write failure only after confirming the original completed action under its own identity, establishing that the checkpoint had not been written, and preserving cancellation and lease checks. The next tool dispatch still goes through its own required persistence and admission. If the checkpoint may have been written but its acknowledgment was lost, the outcome remains unknown; missing receipts or an expired lease are not grounds to continue protected effects. In controlled verification, the first read was not repeated and an independent read and finalization continued. This does not guarantee recovery from arbitrary crashes.
8. Observe disconnection, cancellation, and delivery separately
Closing a browser page leaves an observation channel; pressing Stop expresses cancellation. These are different events. Reconnection should read the original task rather than start a new investigation simply because a subscription reopened.
Read the relationships in groups; the complete diagram remains available below.
- → Detached
- → Settled
- → CancelRequested
- → Running
- → Settled
- → ReplayOriginal
- → Settled
Complete relationship diagram · enlarge for detail
Detached means the connection left. Settled means the task has recorded its actual outcome. The diagram describes connection versus task lifecycle; it does not promise arbitrary continuation after a server-process crash.
A server-side ending also cannot prove the browser received every passage. Generation, commit, transmission, and client display are separate observations. Failure of passage two must not erase passage one; the existence of passage one does not prove the whole question was answered. An attachment needs its own storage and access outcome.
If the final outcome is partial, the public status must retain that fact instead of carrying forward an earlier optimistic completion label. Publication owns answer revisions; the disconnection and recovery essay explains the finer recovery promises.
The public teaching implementation's v0.20 Run control further separates a correctable step from termination of the whole task. One correction within the original budget is permitted only when the original owner supplies validation evidence for the exact call with no unknown or partial effect. Source-permission failures and arbitrary plugin exceptions do not gain a general retry allowance. A mixed group containing a serial Child is rejected before any member dispatches, with paired results persisted; executing half the group first is not a safe fallback.
Similarly, after Publication accepts the body, an ordinary notification failure cannot revoke it. The application records an unconfirmed notification, while actual cancellation, fencing, or result-integrity failures can still prevent success. This does not swallow every error or equate a local timeout with remote termination. Evidence includes offline regression through the original implementation path and hard-kill process recovery. The additional live B tasks did not naturally trigger correction, so their completion rate cannot stand in for live-model acceptance of that correction path.
9. Bring the principle into the next review
When you encounter a throw, early return, or automatic retry, use a concrete task to ask: which independent results should survive this failure? Who records the final state? Does recovery change the goal or repeat an unknown effect?
Ousterhout's discussion of modularity and information hiding emphasizes keeping callers from duplicating internal knowledge. The direction here is similar: decisions belong with the facts needed to make them, not merely in a larger exception-handling function.
For the three documents, the judgment is straightforward: compare what was read, retain the missing part explicitly, preserve saved work, and stop when continuing is unsafe. A module may say “I cannot do this step.” That does not automatically mean “the entire task has no remaining value.”
See principle 4: local failure scope and method 2: responsibility and rejection review. For further investigation, read failure diagnosis and testing and evaluation.
More essays are available at the ETF Agent reading site. Examples and counterarguments are welcome through the GitHub collection; consider a Star if it helps.