A job that fails is loud. A job that stops running is silent.
Alert on the presence of expected output, not on job failure. That one inversion is the cheapest monitoring upgrade I know, and I now build it into every scheduled thing I own — because a job that fails is already screaming at you, and a job that quietly stops being scheduled makes no sound at all.
The evidence that made me hold this rule rather than merely agree with it: a collector in my estate stopped running for 558 consecutive runs across 20 days. Nothing failed. No alert fired, no dashboard turned red, no exit code was non-zero anywhere. There was simply nothing, and nothing is not a thing your monitoring is watching for.
Why failure alerting cannot catch this
Trace what your alerts are actually bound to. Almost every scheduled-job alert you have ever configured is a function of an exit status — the job ran, it returned something, and you evaluate that something. It is a rule about a run that happened.
A job that is not scheduled produces no run. No run means no exit status, no log line, no duration, no error. Every input your alert consumes is absent, so the rule never evaluates, so it never fires. Your monitoring is not broken and it is not misconfigured. It answers "did the last run fail," correctly, forever, about a last run that is receding into the past.
The absence of a signal is not itself a signal — unless you deliberately make it one. That is the whole move.
Write a receipt, alarm on the gap
Have every run write a small dated artifact on success. A JSON file named for the period it covers is plenty:
reports/_receipts/2026-09.json
{"period": "2026-09", "written_at": "2026-09-01T04:00:11Z", "items": 61}
Then point your alert at the receipt, not the job. The rule becomes "a receipt for the current period should exist by 04:30; page me if it does not." Now the failure mode that was invisible has a positive signature, and you get something else for free: the alert stays correct if you migrate the scheduler, rename the job, or move it to a different host. It is bound to the outcome you care about rather than to the mechanism that happened to produce it.
Ask the uncomfortable question about any pipeline you own. If this silently stopped tonight, what would tell me, and how long would it take? If the honest answer is "someone eventually notices the data looks stale," you have a 558-run problem waiting to happen.
Deliver the report; don't make people visit it
The same reasoning changed how I ship my weekly numbers. A dashboard is a thing you have to remember to open, and "I haven't looked in a while" is indistinguishable from "there was nothing to see." So the weekly digest is delivered — it arrives as a chat message, Mondays at 08:00, with the timezone named explicitly in the schedule so it does not drift by an hour twice a year when the clocks move.
Two details do the real work there, and both are absence-shaped:
- A delivery rejection exits non-zero. If the chat platform refuses the message, the job fails loudly instead of completing successfully while nobody receives anything. Without that, a digest that never arrives looks exactly like a quiet week — which is the single most expensive status a report can have.
- The digest reads the same views as the dashboard. Not a parallel query that computes the same idea. The same views, so the two surfaces cannot quietly disagree about a number and leave you arbitrating between them months later.
I hit the drifted-number version of this when I measured who was actually reading this site and found 82% of the traffic was bots. The reported figure and the real one had been separating for a long time with nothing anywhere complaining about it.
Enforcement your plan cannot see will vanish without a sound
Here is the sharper version of the same class, and it is the one I would most want a reader to take away.
I needed a retention rule on one prefix of an object store: keep the recent sweeps, expire the old ones. The storage layer supports exactly that, and applying it took one command. It works. I probed it and confirmed the rule is live.
But that command writes into the storage layer's own metadata store — not into the configuration my infrastructure-as-code manages. Which means the retention rule is completely invisible to plan. My IaC will report no drift, forever, on a control it does not know exists. Rebuild that metadata store during unrelated maintenance and the rule is gone, with every plan still printing a confident no changes.
That is the same failure this whole post is about, relocated one layer down: a control that stops being enforced with nothing going red. So the rule I hold now is blunt. If your plan output cannot see a control, that control can disappear silently, and it needs its own detection — a probe that asserts the rule is still live, alarming on absence, exactly like the receipt.
Two traps worth knowing before you set one of these
Retention often applies at write time, not at read time. Objects written before the policy existed never acquire it retroactively — they simply live forever, invisibly exempt. My prefix happened to hold zero objects on the day I applied the rule, which was the only genuinely clean moment there would ever be for that prefix. Order is the whole safety margin here: set the policy before the prefix accumulates, or accept that you now have two populations and only one of them is governed.
Probe the live value; do not trust your reading of the syntax. The duration format on my store is a number and a unit, and the number is capped at 255. I wanted 397 days. 397d is rejected outright. 13M is accepted — and a month there counts as exactly 30 days, so the rule actually in force is 390 days, not 397 and not 395. I know that because I queried the live setting and compared it against the level above, where it reads zero. Every one of those specifics would have been a guess if I had stopped at the documentation.
Three checks for this week
The key takeaway: your monitoring almost certainly watches for bad outcomes and not for missing ones, and the missing ones are the expensive class.
- Pick your most important scheduled job and disable its schedule. Not the job — the schedule. Time how long until something tells you. If the answer is never, you have found your next hour of work.
- Check whether any report you rely on can fail to arrive silently. A delivery that gets rejected must exit non-zero, or an empty inbox reads as good news.
- List the controls that live outside your IaC. Anything applied by a one-off command to a system's internal state is invisible to your plan and needs its own liveness probe.
Think of it like a smoke detector that reports faults perfectly and has no battery. Every diagnostic it gives you is accurate. It is just not going to be the thing that wakes you up.
If you have caught one of these — a job that had quietly stopped, and something other than luck is what surfaced it — I would really like to hear what tipped you off. I suspect the honest answer for most of us is a customer. As always, I'm here to help!