Our summer shutdown ran for about two weeks across several different production schedules. Most of the work went as planned. One reboot didn’t. We didn’t realize we had a missed reboot until a temp folder problem that should have disappeared after remediation kept coming back.
The hard part of the shutdown was not the patching. It was confirming that every planned step had actually been done.
The window and the scope
The shutdown work ran from June 26 through July 10. There was no single stop date. Each business unit had its own production stop schedule, and most stops lasted between 5 and 10 working days. Public holidays fell inside the window, so some of the work had to be done on weekends.
The scope covered:
- Windows remediation
- Linux remediation
- VMware Tools upgrades
- NetBackup offsite backup validation
- Failover testing for redundant systems
- Disaster recovery testing
We weren’t trying to fit all of this into one maintenance window. Each business unit stopped on its own schedule, so the work had to be coordinated across those different windows.
Why the order looked the way it did
We ran the work in this order:
- Windows patching
- VMware Tools upgrades
- Reboots
- Validation
- Failover and DR-related testing
The order followed the reboot windows. Reboots took the most time, and rebooting was the point where our work met the production schedule. After a server came back, we could check some things ourselves, but application validation depended on the business users who owned the workload. Scheduling those validations ended up deciding the rest of the sequence.
Who validated what
Validation was split in two. Application validation was handled by the business teams. Infrastructure validation was our responsibility.
On the infrastructure side, we checked three places:
- Zabbix
- vCenter
- NetBackup Console
The evidence of the missed reboot was already in Zabbix. When a server reboots, its IPMI signal drops and comes back, and that history should have been in the log. But those events trigger and clear within moments, and across that many servers they piled up fast, so nothing stood out at the time. I didn’t think to check for it.
The reboot we missed
On one server, the patching was finished and the reboot was not. One reboot had been missed.
The omission was not obvious at the time. The clue came later, when a temp folder issue we expected to disappear after remediation kept happening. The server stayed in a partly remediated state until we identified the missing reboot and completed it. After that, the problem went away.
I covered the Windows-side investigation separately in The Temp Folder That Kept Coming Back. This post is about the other half: how one skipped step got through a shutdown that had a plan, a checklist and validation checks.
If we did it again
I had a checklist. The problem was that I was the one holding it.
With several people working across different shutdown windows, that wasn’t enough. The next version needs to be shared, so that whoever performs a task also marks it complete. Every server follows the same sequence, and every step has an owner and a status:
- Install. The remediation and, where scheduled, the VMware Tools upgrade.
- Reboot. This gets its own line for every server, so a missing reboot shows up on the checklist.
- Verify. Infrastructure validation (Zabbix, vCenter, NetBackup Console) is ours, and application validation stays with the business teams.
Next shutdown, a server won’t be considered finished because the patch job says complete. It will be finished when every required step, including the reboot and post-reboot validation, has an owner and a check mark beside it.