When I first saw the proxy log, I was convinced I was looking at a MySQL problem. I wasn’t. The chain that took my Zabbix proxy down that night started with Ubuntu unattended-upgrades.
I didn’t catch this one through an alert. Zabbix can’t very well page you about its own proxy going dark, and the dashboard for this particular environment isn’t one I manage day to day — it came up during a routine round of checking proxy status across the environments I keep an eye on. One proxy simply wasn’t reporting.
The proxy’s own log made the symptom look obvious: cannot connect to database, followed by database is down: reconnecting in 10 seconds, repeating every ten seconds without fail.
Ruling Out MySQL
My first assumption was MySQL — a crash, corruption, connection exhaustion, maybe the disk finally filling up. I went through the checklist: df -h showed plenty of room, free -h showed no memory pressure, SHOW ENGINE INNODB STATUS came back clean, SHOW PROCESSLIST showed nothing stuck. Nothing here explained why the database wasn’t there.
The Clue in the MySQL Log
The real answer was sitting in MySQL’s own error log: Received SHUTDOWN from user <via user signal>.
That line changes the whole investigation. A crash leaves a mess — a stack trace, an OOM kill in dmesg, a corrupted table InnoDB has to recover on the next start. This wasn’t any of that. MySQL had been asked to shut down, and it had done exactly what it was told, cleanly.
Tracing It to systemd
With MySQL ruled out as the cause of its own disappearance, I looked at what else was happening on the box around that time. The system log had the answer: apt-daily-upgrade.service had triggered a systemd reexec, and right behind it came a string of stop commands, all in the same second — mysql.service, zabbix-proxy.service, zabbix-agent.service, and services with no real connection to a package update at all: open-vm-tools, snmpd, ssh.
Ubuntu unattended-upgrades had kicked off its daily run, systemd re-executed itself as part of that process, and that reexec stopped services it had no real reason to touch:
unattended-upgrades → apt-daily-upgrade → systemd reexec → mysql.service stopped → zabbix-proxy stopped → no automatic restart → monitoring outage.
Timeline
| Time | Event |
|---|---|
| 06:31:57 | apt-daily-upgrade.service starts |
| 06:32:35 | systemd reexec requested |
| 06:32:37 | systemd reexecutes |
| 06:32:37 | mysql.service stops |
| 06:32:37 | open-vm-tools, snmpd, ssh also stop — same second |
| — | Zabbix proxy log fills with database is down: reconnecting in 10 seconds |
| Several hours later | Outage found during a routine proxy review |
| — | MySQL restarted manually |
| — | Proxy reporting resumes |
Why It Didn’t Come Back On Its Own
This is the part that made it worse than an ordinary crash. Because MySQL exited cleanly — a normal shutdown, not a failure — systemd’s Restart=on-failure setting never triggered. on-failure only reacts to a non-zero exit or a signal it treats as abnormal. A graceful stop doesn’t count, even if nothing asked for it on purpose.
So MySQL just sat there, stopped, for several hours, until a routine proxy check was what caught it — not anything coming from the environment itself.
Lessons Learned
A clean shutdown and a crash leave completely different evidence, and only one of them gets an automatic restart. If a service exits successfully, systemd has no reason to think anything is wrong — even when no human actually asked for that shutdown.
A proxy problem isn’t always a proxy problem. Everything pointed at Zabbix and MySQL until the actual chain of events turned out to start with the OS’s own update schedule.
The proxy didn’t fail. It lost the database it depended on.
Preventive Actions
Since then, systemctl is-active mysql and systemctl is-active zabbix-proxy are things I check directly rather than inferring from whether Zabbix itself looks healthy — that doesn’t work when the proxy is the thing that’s down.
I also went back and looked at how unattended-upgrades is configured on boxes like this one. There’s no clean universal answer between Restart=on-failure and Restart=always here — on-failure avoids restart loops when a service is failing for a real reason, but it also means a clean shutdown like this one goes unnoticed until someone checks. Restart=always closes that gap but can mask other problems by restarting straight through them. Which one makes more sense depends on what’s running and how bad a silent outage is compared to a silent restart loop.
Frequently Asked Questions
Why didn’t systemd automatically restart MySQL after it stopped?
Because MySQL exited cleanly. systemd’s Restart=on-failure only restarts a service after an abnormal exit — a crash, a non-zero exit code, a signal it treats as a failure. A graceful shutdown, even an unplanned one, doesn’t meet that bar, so nothing brought it back up.
How can I catch this kind of outage faster next time?
Check the service directly rather than relying on the tool to report on itself. systemctl is-active mysql and systemctl is-active zabbix-proxy will tell you the truth even when Zabbix’s own alerting has nothing to say — because the thing that would normally raise the alert is the thing that’s down.
Related reading:
- The instinct to trust that “no alert” means “no problem” is exactly what The Alert That Never Fired was about, just from the other direction — that time, a threshold never fired even though something was wrong; this time, nothing could fire because the thing meant to report was the one that went down.
- Ubuntu’s own update schedule isn’t the only kind of maintenance that can quietly take a production service down without asking — see Windows Server Patch Tuesday for the same lesson on a different OS.