Recovery Is a State Decision, Not an Undo Button

A Transient Error Is Not Permission to Retry

Consider an unattended backup command that times out after it may already have written an archive. Calling the error transient makes a retry sound reasonable. But the process error does not answer the questions that matter most:

• Did the task emit the receipt that proves which attempt ran?
• Did cleanup finish, or is an old effect still active?
• Did the privacy checks complete?
• Is the semantic result known, or merely the process exit?
• Does this controller still own the right to act?

If any required answer is missing, “retry the transient failure” is no longer a recovery policy. It is a guess that may duplicate an effect, hide incomplete cleanup, or let a stale owner act.

Recovery disposition should come from independent evidence axes before the controller interprets the execution error.

This does not mean every failure needs a human. It means automatic remediation is reserved for the smaller set of failures whose required evidence is complete and whose transient class has a named, bounded handler.

Five Evidence Questions, Then One Decision

manual_review is a disposition for one run whose evidence needs human judgment. Circuit-open is the policy-level refusal to start more automatic attempts until the unsafe condition is cleared. Both stop remediation, but they describe different scopes.

The concrete objects matter here. A receipt gap, a cleanup gap, and an ownership gap are not interchangeable “boundaries.” Each removes a different fact the controller would need before acting.

What the Offline Reproduction Actually Showed

I exercised the rule in a disposable SQLite-backed reproduction with three terminal observations. An adapter translates one task type's terminal record into the controller's generic observation shape. Two observations belonged to a generic backup adapter: one clean success and one verified transient dependency failure. A different task type omitted evidence required by its policy and therefore stopped for review. No live external delivery or production state was involved.

The four records are transition notifications captured by an effect-free offline handler, not four remote messages. The replay result is narrow but useful: these three replayed observations produced no second transition notification inside this controller. It does not turn an arbitrary external API into an exactly-once system.

The State Machine Replaced My Pile of Undo Commands

The original version of this article proposed a standard collection of --undo, --replay, and --rollback flags. Those can be useful interfaces, but they do not decide whether an operation is safe to perform. The controller now makes that decision first:

def decide(observation, policy):
    gaps = required_evidence_gaps(observation, policy)
    if gaps:
        return manual_review(reason=gaps)

    if observation.result == "success":
        return completed()

    handler = policy.named_handler_for(observation.failure_class)
    if handler and handler.is_verified_transient:
        reserve_effect_once(observation.operation_key)
        return run_with_attempt_cap(handler, policy.max_attempts)

    return circuit_open(reason="no safe automatic handler")

The important order is evidence, disposition, reservation, then effect. An error classifier cannot jump directly to the final line.

State, Transition, and Notification Move Together

A recovery decision is not durable if the database says recovered but the transition event is missing, or if a notification can be produced twice after a crash. The implementation writes three things in the same transaction:

1. the current run state;
2. the transition event that explains how it changed;
3. an owned outbox record for the notification that still needs delivery.

A stable operation key is the durable identity used to find the already-recorded decision on replay. In this reproduction, a duplicate observation found the existing state and outbox lineage instead of inventing a second recovery attempt.

The outbox does not make a remote receiver transactional with SQLite. If a send times out after the receiver may have accepted it, the correct state is still “ambiguous.” The controller reconciles with a supported read-back or waits for a human decision; it does not manufacture certainty by sending again.

The Tradeoff I Chose

I would rather route an unusual task to manual review than broaden the automatic-remediation set around an attractive error label. That choice gives up some unattended recovery rate. In return, missing receipts, incomplete cleanup, privacy failures, unknown results, and stale ownership cannot borrow the word transient as permission to act.

Automatic recovery earns its place one handler at a time:

• the failure class is explicit and tested;
• the policy names the handler rather than discovering it dynamically;
• an attempt cap is part of the policy;
• the remediation action is reserved before that handler executes;
• replay of the covered observations produces no duplicate action.

What This Proof Does Not Establish

This is a sanitized offline implementation, not an upstream OpenClaw feature or a universal recovery standard. It does not prove live external delivery, hostile same-user integrity, crash safety for arbitrary side-effect handlers, production load behavior, or that every transient failure is safe to remediate.

It would also fail its own claim if a required evidence gap could enter automatic remediation; if replay created another state transition, handler attempt, or notice; if ambiguous delivery triggered an automatic resend; or if a handler could run without its named policy, attempt cap, and pre-effect reservation.

A More Useful Recovery Checklist

I still want snapshots, reversible operations, dry runs, and restore procedures. I now ask these questions first:

• Which evidence axes are required for this task?
• Which missing axis forces manual review or circuit-open?
• Which exact transient classes have named remediation handlers?
• Where is the pre-effect reservation made?
• Did the bounded replay test create another transition, handler attempt, or notification?
• How is an ambiguous external effect reconciled without blind retry?

An undo command answers “how could I reverse this?” A recovery controller must answer the earlier question: “given what I can prove, am I allowed to do anything at all?”