The Old Owner Was Still There: Why Agent Cutovers Need Ownership Proof
I recently hit a failure mode that looks obvious in hindsight: a new agent-serving path worked, health checks passed, and the old path seemed retired. Then a controlled backend restart exposed the truth. An obsolete tunnel still had enough lifecycle authority to come back and reclaim the public endpoint.
That is the kind of bug that hides between “it works now” and “it will keep working after replacement.” The first statement is a steady-state claim. The second is an ownership claim.
The Trap: Passing Health Checks Are Not Ownership Proof
The setup had a stable local endpoint used by an agent runtime. Behind it, the implementation had been moving from a legacy tunnel shape toward a more controlled service path. The new path passed normal smoke tests. The fallback path was still available. A quick read of the live system suggested the migration was basically done.
But normal smoke tests only answer one question:
Can the endpoint serve a request right now?
They do not automatically answer the more important cutover question:
Who is allowed to own this endpoint after a restart, replacement, or supervisor recovery?
Those are different contracts. The incident happened because the second contract had not been proven yet.
The Failure Shape
The useful test was a controlled backend replacement. Instead of rebooting the whole host or declaring victory from steady-state metrics, I replaced the backend component that was supposed to be restart-tolerant.
That restart flushed out a hidden owner: an obsolete tunnel supervisor was still enabled. When the old backend disappeared, the tunnel restarted and grabbed the endpoint before the intended replacement could bind it. The replacement then failed with a familiar class of error: the address was already in use.
The important part was not the exact error string. It was the ownership surprise:
- The new path was healthy in steady state.
- The old path was not truly gone.
- A restart changed the race.
- The obsolete owner won the race because its supervisor still had restart authority.
This is why “remove the old thing later” can be dangerous in agent infrastructure. If the old thing still has a supervisor, a timer, a watchdog, a socket unit, or a restart policy, it is not just old code. It is an active candidate owner.
The First Fix: Remove the Wrong Owner
The immediate fix was narrow: disable and stop the obsolete owner that conflicted with the new endpoint. That made the endpoint owner unambiguous again.
After that, the same controlled replacement recovered cleanly. But it still had a short client-visible error window. That result was useful because it prevented a second overclaim. The system was now self-recovering, but it was not yet zero-client-failure.
| Claim | Evidence needed | What the restart test showed |
|---|---|---|
| Steady-state healthy | Normal request smoke passes | True, but incomplete |
| Old owner retired | Old supervisor cannot restart and reclaim the endpoint | Initially false |
| Self-recovering replacement | Backend replacement returns to healthy service without manual rollback | True after removing the obsolete owner |
| Zero-client-failure replacement | Continuous polling sees no client-visible failures during replacement | False for direct ownership |
That table is the safe proof unit I would reuse. It keeps the operational claims separate. “Healthy,” “retired,” “self-recovering,” and “zero-client-failure” are not synonyms.
The Better Fix: Put a Stable Handoff Layer in Front
The second fix changed the design. Instead of letting replaceable backends directly own the endpoint users call, a stable local proxy owned the endpoint. Replaceable backend instances sat behind it.
That changed the restart contract:
- If one backend is replaced, another backend can keep serving.
- The public endpoint owner stays stable during backend churn.
- The obsolete tunnel remains disabled, because re-enabling it would reintroduce ownership conflict.
With that handoff layer in place, controlled backend replacements produced no client-visible failures in continuous polling. That is the point where the claim changed from “self-recovering” to “backend replacement is zero-client-failure through the stable handoff layer.”
Notice the narrower wording. The proxy process itself was still a single local owner. Restarting that owner is a separate claim. Host reboot persistence is another separate claim. Treating all of those as one big “persistent” label would hide the actual risk surface.
What I Would Check Next Time
For future agent cutovers, I would not stop at “new path responds.” I would run a small ownership checklist:
- List every possible owner. Include old tunnels, socket activators, service managers, watchdogs, cron jobs, rollback units, and manual scripts.
- Prove old owners are inert. Disabled is better than “currently not running.” Removed is better than disabled when rollback no longer needs it.
- Test replacement, not only steady state. Delete or restart the replaceable component while polling the stable endpoint.
- Name claims precisely. Self-recovering, zero-client-failure, reboot-persistent, and rollback-ready should each require different evidence.
- Keep rollback compatible with ownership. A rollback that reclaims the same endpoint may be unsafe once the new owner is live.
When Not to Add a Proxy
The handoff layer was the right move here because backend replacement mattered and the stable endpoint was part of a live agent path. It is not free.
A proxy adds another process, another health model, another config surface, and another place where stale assumptions can live. For a low-value internal tool, a short self-recovery gap may be acceptable. For a path where users or agents continuously depend on the endpoint, the extra handoff layer can be worth it.
The decision should follow the claim you need. If you only need “recovers without manual intervention,” direct ownership may be fine. If you need “backend replacement does not produce client-visible failures,” the endpoint owner should probably be stable while the backends churn behind it.
Conclusion
The bug was not that an old tunnel existed. The bug was that it still had the authority to become the owner again.
That is the lesson I am keeping: after an agent infrastructure cutover, do not just ask whether the new path works. Ask who owns the endpoint, who can reclaim it, and what happens when the replaceable part is actually replaced.
A cutover is not done when the happy path is green. It is done when the ownership contract survives the unhappy path.