My Proxmox backups had been failing silently for ten days. Not failing, actually. Not running at all, and nothing alerted on it.
The daemon that quietly stopped
A cluster shakedown turned up pvescheduler.service sitting inactive (dead) on one node since the last upgrade window. That daemon runs every scheduled backup, replication, and prune job on the host. When it is stopped, the daily job does not error. It does not fire. There is no failed task in the log, because no task was ever started. The system was, from its own point of view, perfectly healthy. It just was not doing the one thing I was counting on.
The gap that let it hide
Most monitoring checks systemctl is-active. That catches the state where a service is supposed to be running and has crashed: it was up, now it is down, page someone. What it does not catch is the state where a service is supposed to be running and was cleanly stopped and never came back. To is-active those look different, and the second one is where silent failures live.
The Prometheus rule that would have paged me on day one is short:
node_systemd_unit_state{state="active"} == 0
and node_systemd_unit_state{state="enabled"} == 1
Enabled but not active. The unit is configured to run at boot, and right now it is not running. That combination is almost always a mistake, and it is the cheapest dead-daemon check there is. It would have caught this on the first morning instead of the tenth.
The fix, and the lesson
The fix once I found it took ninety seconds: systemctl start pvescheduler.service, and the next backup window ran all eleven guests. The fix is never the interesting part of one of these. The interesting part is the ten days.
is-active is table stakes for monitoring; everyone has it. enabled-but-inactive almost never is, and that gap is exactly where the quiet failures sit. If you run scheduled backups, go check right now that they actually ran recently, and then add the alert that would have told you without you having to look.
