Around 4:37 a.m. on June 25, 2026, the primary database server behind our manufacturing execution system crashed with a blue screen. Database server failover to the standby completed within a minute, and production kept running.
This post covers the timeline, what the crash dump pointed to, and a decision I found more interesting than the failover itself: we didn’t fail back right away.
The Setup
The database was Microsoft SQL Server on a two-node Active/Standby cluster. The HA software watches both nodes through a heartbeat and a dedicated HA link, and if the active node fails, the standby takes over the SQL resource group automatically.
I haven’t verified how data is kept in sync between the two nodes, so I’m leaving that out rather than guessing.
The Timeline
All times are from the morning of June 25, 2026. Entries without seconds are approximate.
| Time | What happened |
|---|---|
| ~04:37 | Primary database server blue-screens and shuts down abnormally |
| 04:37:21 | A replication socket error is logged |
| 04:37:43–47 | The HA link changes to FAULTED |
| 04:37:47 | The standby starts taking over the SQL resource group |
| 04:37:58 | Takeover succeeds, SQL resource group is ONLINE on the standby |
| ~04:40 | Primary boots back up |
| ~04:41 | Primary reboots again, this time for Windows servicing |
| ~04:44 | Primary finishes booting normally |
| 04:44:57 | HA heartbeat goes from UNSTABLE back to STABLE |
The takeover itself took about eleven seconds, from the moment it started (04:37:47) to the SQL resource group coming online (04:37:58). Counted from the first error in the logs, it was still well under a minute.
Production Continued on the Standby
The primary went down hard, but the standby took over the SQL resource group and stayed online. The production floor reported no impact, and there was no production interruption.
What the Crash Dump Said
The vendor analyzed the minidump from the crash. The key fields:
BUGCHECK_CODE: 133
DPC_TIMEOUT_TYPE: SINGLE_DPC_TIMEOUT_EXCEEDED
FAILURE_BUCKET_ID: 0x133_DPC_IPMIDrv!KCSDpc
MODULE_NAME: IPMIDrv
IMAGE_NAME: IPMIDrv.sysBugcheck 0x133 is the DPC watchdog. When a deferred procedure call runs longer than Windows allows, the system assumes it’s hung and stops on purpose. The vendor’s minidump analysis associated the timeout with IPMIDrv.sys, the driver Windows uses to talk to the server’s management controller.
That gave us a lead, but it didn’t establish the underlying defect, and I haven’t seen Microsoft confirm one.
What We Did After the Crash
Following the vendor analysis, we applied the available Windows security updates and the relevant driver updates, then kept watching the server. The same BSOD hasn’t recurred since. I can’t say for certain the updates were the fix, only that it hasn’t come back.
The failback is the part I think is worth the space.
By about 04:44 the primary was healthy again. But production was already running on the standby, and taking the workload back meant interrupting the database service that was serving it. So we left it there. The standby stayed Active and the primary stayed in the standby role.
The production schedule kept running through the weekends, so there was no gap in which we could take the database down. The first real window came two weeks later, with the summer shutdown. During that shutdown we moved the workload back to the original server, and the Active/Standby layout was restored. (More on that shutdown in the Summer Shutdown write-up.)
What I Took From This
High availability didn’t prevent the server from crashing. It kept one server crash from becoming a production outage. And once the crash was contained, there was no reason to add a second database interruption before we had a proper window for it.
Related reading: The Alert That Never Fired covers a different gap: monitoring that looked fine until it was needed.
Frequently Asked Questions
What is Bugcheck 0x133 on Windows Server?
It’s the DPC watchdog violation. Windows raises it when a deferred procedure call, or the system running at an elevated interrupt level, exceeds its time limit. Drivers are a common suspect, and the failure bucket in the dump usually names the module involved. In our case the vendor’s analysis named IPMIDrv.sys.
Should you fail back immediately after a failover?
Not necessarily. If production is stable on the standby, a failback is one more interruption to a service that’s currently working. We waited for a planned maintenance window instead.