The failure looked like a failed automation job. A daily scan was supposed to collect a small digest and post it to a configured output channel. Instead, the visible message did not land.
The important detail was easy to miss: the report itself existed. The collection and summarization path had completed. The broken piece was the final delivery side effect.
Delivery failure is not automatically generation failure.
The Failure Shape
The incident had three separate surfaces:
| Surface | What mattered | Interpretation |
|---|---|---|
| Generated artifact | The saved digest existed and contained the expected summary. | The data collection and summarization lane was probably healthy. |
| Delivery state | The configured output channel did not receive the final message. | The last-mile delivery side effect needed investigation. |
| Sibling lanes | Some nearby automations delivered normally while another delivery-oriented lane also showed transient trouble. | The provider surface looked flaky, not uniformly broken. |
That split changed the response. If the artifact had been missing, the right move would have been to debug collection, credentials, source API access, or the summarizer. But with the artifact already present, a full rerun risked creating duplicate work, different output, or confusing evidence.
The Triage Ladder
The useful ladder was deliberately boring:
- Check whether the artifact exists. If yes, preserve it and treat it as the source of truth for recovery.
- Check whether delivery actually completed. Do not infer success from generation logs or from the job reaching the final step.
- Compare neighboring lanes. If unrelated delivery lanes are also flaky, suspect the shared delivery surface before rewriting local config.
- Check external provider health. A transient message-send incident can look like a local automation regression if you only stare at your own logs.
- Replay the saved output when safe. If the report is already generated and public-safe for its intended destination, replaying is often better than rerunning the collector.
This is the same reliability habit I want in larger agent systems: distinguish the durable artifact from the side effects around it.
Why Rerunning Was the Wrong First Move
Rerunning feels productive because it gives the operator a new attempt. But it also changes the evidence.
| Move | Risk | Better default |
|---|---|---|
| Rerun the whole scan immediately. | Duplicate collection, drifted summary, or a second delivery attempt that hides the original failure shape. | Replay the already-saved digest if it is complete. |
| Patch local config during the incident. | Fixes the wrong layer when the shared provider is briefly unhealthy. | Check provider health and sibling delivery lanes first. |
| Declare the job broken. | Mislabels a last-mile delivery failure as a generation failure. | Name the failing surface precisely. |
The outcome was intentionally unglamorous: the saved digest was replayed to the configured destination, the local workflow was left unchanged, and the next natural run was allowed to prove whether the issue repeated.
The General Agent-Ops Lesson
Agent workflows often mix three things that deserve separate status:
- artifact generation — did the workflow create the report, summary, file, or decision packet?
- state recording — did the system record enough evidence to recover or audit the run?
- external delivery — did the final message, post, email, or notification reach its destination?
When those are collapsed into one success bit, operators make bad choices. A failed delivery can trigger a needless rerun. A successful generation can look like a total outage. A transient provider incident can send you hunting through local code that did nothing wrong.
A Small Playbook I Would Reuse
For any unattended agent job that posts somewhere public, semi-public, or team-visible, I would want this playbook:
- Artifact-first execution. Write the report before trying to send it.
- Separate delivery status. Store whether the message reached the configured destination.
- Idempotent replay. Make it easy to resend the saved output without recollecting data.
- Sibling-lane comparison. Before changing local settings, check whether other jobs using the same delivery surface were also flaky.
- External incident check. If the error smells like upstream availability, check provider health before assuming local drift.
None of that is complicated. That is why it is useful. The goal is not a heroic recovery story; the goal is to make the boring recovery path obvious while the incident is still fresh.
Conclusion
The lesson from this incident is not that daily scan jobs are fragile. The lesson is that delivery is a separate operational surface from generation.
If the artifact exists, preserve it. If the delivery surface is flaky, diagnose that layer directly. If the output is already complete, replay it instead of rerunning the whole workflow and manufacturing new uncertainty.
That one distinction turns a vague failed-job alert into a clean agent-ops decision: recover the visible output, avoid unnecessary local changes, and only reopen the system if the same failure repeats outside a known external incident window.