A control-plane monitor fired because one session-store metric crossed a historical row-count threshold.
The alert was not fake. The count really had crossed the line. But the operational question was different: did that threshold crossing mean the system needed intervention?
A true alert can still be the wrong page.
The Failure Shape
The monitor was watching session-store pressure. Historically, row count had been a useful proxy: more rows often meant more work to list, filter, render, or clean up. That made a row-count threshold a reasonable early guardrail.
But after cleanup and compaction improvements, the meaning of a row changed. The store could contain more rows while still staying small on disk and acceptable in latency. A threshold calibrated for older row shapes had become too sensitive.
| Signal | What it said | Operational read |
|---|---|---|
| Row count | A historical count threshold had been crossed. | Worth recording, but not enough by itself to interrupt the operator. |
| Byte pressure | The store was still modest in size. | No evidence of storage bloat or urgent compaction need. |
| Latency spot checks | Interactive checks were not showing a severe regression. | No immediate user-facing control-plane pressure. |
| Cleanup candidates | Dry-run cleanup evidence did not show enough reclaimable state to justify a risky operation. | Cleanup would add lifecycle risk without enough expected benefit. |
That combination changed the right action. If byte pressure, latency, and cleanup candidates had all agreed with the row count, the alert would have deserved escalation. But when only the row count moved, the better conclusion was narrower: the monitor needed to preserve the observation without treating it as a page-worthy incident.
Why the First Fix Was Not Cleanup
The tempting response to a storage-shaped alert is to run cleanup. In an agent gateway, that can be the wrong instinct.
Cleanup is not free. Depending on the system, it may touch active state, registry records, session history, or service lifecycle boundaries. Even when a cleanup script has a dry-run mode and conservative filters, running it during a weak alert can turn a harmless threshold crossing into a real incident.
So I treated the alert as a diagnosis problem first:
- Confirm the metric. The row-count crossing was real.
- Check pressure dimensions. Size, latency, and cleanup-candidate evidence did not support urgent action.
- Avoid lifecycle mutation. No service restart, destructive cleanup, or active-state rewrite was justified.
- Tune the monitor. Keep row-only crossings visible in details, but alert only when pressure is multi-dimensional or clearly extreme.
The Pattern: Alert on Pressure, Not Telemetry Alone
This is the broader design lesson. A metric crossing a threshold is telemetry. An alert should mean the system needs attention.
Those are not the same thing.
| If the monitor sees… | Then treat it as… | Default response |
|---|---|---|
| One weak proxy metric crosses a historical line. | Telemetry. | Record it; do not page unless it persists or accelerates. |
| Count and byte pressure rise together. | Real pressure candidate. | Alert and investigate cleanup or compaction options. |
| Count rises and latency regresses. | User-facing risk candidate. | Alert and prioritize performance diagnosis. |
| Cleanup dry-run shows substantial reclaimable state. | Maintenance candidate. | Schedule the safest cleanup path with lifecycle controls. |
This is especially important for self-hosted agents because the operator is often the same person who receives the alert, investigates it, approves risky actions, and deals with the fallout. Alert noise is not just annoying; it burns the judgment budget you need for the alerts that are real.
The Monitor Change
The monitor became more honest after the change:
- Row-only crossings are still recorded. They remain visible for trend analysis and future calibration.
- Alerts require stronger evidence. Byte pressure, combined count-and-size pressure, latency impact, cleanup pressure, or a high row-only guardrail can still trigger action.
- Risky maintenance stays gated. Cleanup remains something to justify, not something to reflexively run because one proxy moved.
The important part is not the exact threshold. The important part is that the alert now reflects operational pressure rather than a stale proxy.
A Small Checklist I Would Reuse
For any agent control-plane monitor, I would reuse this checklist before turning a metric into a page:
- Name the pressure dimension. Is this size, latency, error rate, freshness, cost, queue depth, or merely count?
- Ask whether the metric is still a proxy. If the system changed, old thresholds may no longer mean what they used to mean.
- Require corroboration for risky actions. Do not run cleanup, restarts, or state rewrites from one weak signal.
- Keep demoted signals visible. A non-page-worthy signal can still be useful in daily snapshots or trend logs.
- Write the reopen criteria. Decide what evidence would make the next alert actionable.
This makes the system quieter without making it blind.
Conclusion
The lesson from this incident is not that row-count thresholds are bad. They are useful early-warning signals. The problem is treating every early-warning signal as an operator page forever.
As systems mature, old proxies need recalibration. If rows become smaller, row count alone may stop predicting pressure. If cleanup becomes riskier than the condition it addresses, cleanup should require stronger evidence. If a monitor can explain the difference between telemetry and pressure, the operator can make better decisions.
A true alert is only useful when it points at real action. Otherwise, the right fix is not to silence the monitor. It is to teach the monitor what deserves attention.