A Green Scheduled Run Can Still Hide a Broken Inner Producer

One of my unattended agent workflows looked healthy from the outside.

The scheduler had a successful outer result. A core phase still completed useful work. The report itself rendered. But one inner producer had stopped writing the durable artifact that made the workflow complete.

A green envelope can prove that the wrapper finished while saying almost nothing about a required inner producer.

Two Bugs, Only One Broken Producer

The first bug was easy to see: the report parser recognized a date-only label, while the actual artifact used a full timestamp. The reporter could miss a valid entry even when the entry existed.

That was a real reporting defect. It was not the whole incident.

After fixing the parser, the artifact was still stale. The inner producer itself was no longer completing. This distinction changed the repair:

A parser repair can make a report honest. It cannot create output that the producer never wrote.

The Bounded Failure Mechanism

In the installed snapshot I examined, durable conversation pointers were protected from ordinary retention cleanup. That protection was intentional: continuity should not disappear merely because a maintenance pass needed room.

The problem appeared at the boundary. Protected durable pointers had already exhausted ordinary capacity under the retention budget. A newly admitted ephemeral producer session was not yet leased as active work. Maintenance preserved the protected rows and removed the ordinary new row while the producer was starting.

The result was a strange but coherent state:

• the protected conversations survived;
• the outer scheduled path completed;
• a core phase could still report progress;
• the ephemeral producer disappeared; and
• the durable artifact stopped advancing.

A Public-Safe Reproduction

The smallest useful model needs only protected rows, one ordinary producer, a budget, and a bounded lease:

from dataclasses import dataclass

@dataclass(frozen=True)
class Row:
    name: str
    protected: bool = False
    leased: bool = False

def names(rows):
    return {row.name for row in rows}

def enforce_budget(rows, budget):
    durable = [row for row in rows if row.protected or row.leased]
    ordinary = [row for row in rows if not row.protected and not row.leased]
    available = max(0, budget - len(durable))
    return durable + ordinary[-available:] if available else durable

protected = [Row("durable-a", protected=True),
             Row("durable-b", protected=True),
             Row("durable-c", protected=True)]

unleased = protected + [Row("ephemeral-producer")]
assert "ephemeral-producer" not in names(enforce_budget(unleased, budget=3))

leased = protected + [Row("ephemeral-producer", leased=True)]
assert "ephemeral-producer" in names(enforce_budget(leased, budget=3))

This model establishes a bounded mechanism: under protected-state saturation, an unleased ordinary producer can be the only removable row; leasing admitted work preserves it through startup without deleting or unprotecting durable continuity.

It does not establish that a lease is the unique repair, that a global budget change could never help, or that every retention implementation behaves this way.

Outer Success Is One Component, Not the Verdict

The report originally compressed several different facts into one friendly status. The repair I wanted was a vector:

overall = OK only if:
    outer_envelope_ok
    and core_phase_ok
    and inner_producer_ok
    and artifact_is_fresh
    and artifact_matches_sweep
    and same_sweep_receipt_is_terminal

That rule intentionally lets a job be partly successful and globally degraded at the same time. Honest partial failure is more useful than a single green status that erases which component needs work.

Why the Receipt Must Be Sweep-Scoped

Artifact freshness alone is not enough. A recently modified file could belong to an earlier run. A new receipt without an artifact could report intent rather than effect.

The bounded candidate therefore records one sweep identity and moves it once from pending to terminal:

{
  "sweep": "stable-correlation-key",
  "outcome": "degraded",
  "phases": [
    {"name": "core", "outcome": "completed"},
    {"name": "producer", "outcome": "failed", "fallback": 1}
  ],
  "artifactWrites": 0
}

Here, fallback: 1 records one bounded fallback action for the failed producer. It is not a successful write of the required artifact, so artifactWrites remains zero.

The monitor then reconciles four things:

1. the expected sweep identity;
2. one terminal receipt for that sweep;
3. the per-component outcome; and
4. the artifact write/freshness evidence.

A failed producer may emit one bounded fallback, but the terminal receipt is immutable. A second completion attempt must not manufacture a second fallback or rewrite history.

Archive Before Trimming

The same isolated candidate applies one additional durability boundary to the artifact itself: archive the old material before trimming the live file.

This was not the observed startup-eviction root cause. It is adjacent hardening for a producer that may resume writing after a long outage.

if archive_creation_succeeds:
    replace_live_file_with_bounded_tail()
else:
    leave_original_bytes_unchanged()

The public-safe control checks byte identity on archive failure. A retention helper that shortens the source before it knows the archive exists has converted housekeeping into data loss.

Tempting Changes Are Separate Experiments

• Parser-only repair: necessary for truthful observation, insufficient to prove producer recovery.
• Raise the global retention budget: may change pressure, but the public proof does not compare its safety, cost, or long-term behavior.
• Remove durable protection: may create removable rows, but risks trading a producer failure for lost continuity.
• Tune alert thresholds: changes when the system complains, not whether the artifact exists.

I did not need to claim that every alternative fails. I needed one candidate that preserved the boundary I cared about: admitted ephemeral work should survive startup without weakening the durable state it depends on.

When Not to Use This Pattern

• Do not blame retention pressure before reproducing the relevant state shape.
• Do not use a lease as an unbounded pin; hung and crashed producers still need cleanup and terminal failure semantics.
• Do not infer health from a fresh timestamp without binding it to the expected sweep.
• Do not infer exactly-once execution from one immutable receipt; receipt/write atomicity, retry identity, crash windows, and late completion remain outside this proof.
• Do not treat the outer scheduler status as a component-health aggregate.
• Do not claim an isolated candidate is current upstream behavior or a live recovery.
• Do not trim a durable artifact before archive success is proven.

A Small Review Checklist

1. List required components instead of trusting the outer status.
2. Separate parser/readback bugs from producer execution failures.
3. Bind artifact evidence to the same run or sweep.
4. Protect admitted work only for a bounded lifecycle window.
5. Preserve durable continuity; do not “fix” pressure by deleting the evidence you need.
6. Require one immutable terminal receipt and idempotent fallback behavior.
7. Keep live/upstream claims behind actual live/upstream evidence.

Conclusion

The useful lesson was not merely that a green status can lie. It was that the word success belonged to the wrong scope.

The outer wrapper succeeded. One core component succeeded. The inner producer failed, the artifact stayed stale, and the workflow was degraded.

In the isolated offline candidate, component-scoped health and sweep-scoped receipts could tell that truth without sacrificing durable conversation state or pretending the candidate was already live.