The call came in while I was still driving to work. Production line down, database server unresponsive. There wasn’t much I could do from the car except say I was on my way.
By the time I got to my desk, the database was already back. What had actually happened — an MSSQL TempDB full event overnight — was already over. Connections were going through, nothing looked wrong. Which was its own kind of unsettling: something had clearly broken hard enough to shut down a production line, and now there was nothing on screen to point at.
So the work started after the fire was already out.
The Blind Spot I Already Knew About
First place I checked was Zabbix — normally where I’d pull up a timeline and see what tipped over first. For this client, though, the proxy connection had been down for a while. This was the same client whose Zabbix proxy I’d already written about before — the kind of client where it dropped often enough that I’d stopped expecting it to just stay up.
I’d known about that gap. The support relationship with that client sat in a gray area — not clearly anyone’s job to fix — so it had just stayed broken. Usually that didn’t cost anything. This time it meant walking into the investigation without one of my usual tools.
With Zabbix out, I went to Windows Event Viewer and started working backward from when the calls came in.
What the Logs Actually Showed
Around 5:30 AM, the drive holding TempDB filled up completely. The log pointed to active transactions as the reason — enough of them, running long enough, that TempDB kept growing with nowhere left to go.
By around 7:40 AM, the system stopped logging further events on that drive. The event log showed MSSQL stopping around that point — and sometime after, it was back up, TempDB had room again, and everything looked normal.
Nobody restarted it by hand. Whatever brought it back down and up again happened on its own, somewhere in that two-hour window.
Why This Didn’t Turn Into a Bigger Problem
The MSSQL TempDB full event itself didn’t trace back to anything on our side — whatever generated those transactions was upstream of us, not something we’d changed. That’s what kept this from becoming a longer conversation afterward.
The monitoring gap was different. That one actually was ours to own, even with the contract making it easy to leave alone. It just didn’t matter this time, because the outage had already resolved itself before I needed the data I didn’t have.
I didn’t get to fix the TempDB issue — it was gone before I heard about it. The blind spot I’d been aware of and hadn’t pushed on, though, was still sitting there exactly where I’d left it.
The line was back up before most of the plant had even noticed it went down. If the timing had been an hour different — if this had happened during a shift with people already staring at dashboards, waiting on an answer — the missing Zabbix history is the one thing I would have actually needed and didn’t have.
Related reading:
Frequently Asked Questions
What actually caused this MSSQL TempDB full event?
The Windows event logs pointed to active transactions on that server — enough of them, running long enough, that the TempDB drive filled up before anything could catch up. What generated those particular transactions wasn’t something we could trace back to a change on our side.
Why wasn’t this caught by monitoring before it became an outage?
The Zabbix proxy for that client had been disconnected for a while already, tied to how support responsibility was split contractually. It was a gap I already knew about — this incident just happened to be the one where I actually needed the data behind it.