The first time we ran disaster recovery testing here, it didn’t expose a technology problem. It exposed a documentation problem — and that turned out to be the harder one to fix.
The Setup
I started managing this environment in October 2025. The stack is VMware on the virtualization side, Hitachi and Dell Unity on the storage side, and NetBackup handling backup and recovery across both.
Snapshots on our storage arrays only exist if someone manually takes one — there’s no scheduled snapshot policy quietly running in the background. In practice, that makes NetBackup our real recovery path for almost everything, snapshots or not.
Why We Actually Tested It
Two things pushed this from “we should test our DR plan someday” to “we’re testing it this week.” HQ needed audit evidence that recovery actually worked, not just a diagram saying it should. And our summer shutdown window gave us a natural opportunity to run it for real instead of on paper. So we ran our first full test in winter 2025, and a second one in summer 2026.
What the First Test Actually Exposed
The infrastructure did what it was supposed to do. NetBackup restored what it restores. But when we sat down to actually run the test, we didn’t have anything written down about what “recovered” meant for each system — which recovery point to restore to, what scope counted as done, who signed off. That gap, not a technical failure, was what caused the confusion on test day.
What We Built Afterward
Instead of treating that as a bad day and moving on, we used it to write the process that should have existed from the start:
- VMs: restore from the most recent NetBackup version immediately when an incident happens.
- Databases: because of transaction consistency, restore first, then confirm you’ve landed on a version from before the issue occurred, or one with no transaction problems — recovery speed and data integrity aren’t the same checkbox.
- RTO targets, set for the first time: databases within 1 hour, VMs within 10 minutes.
- An emergency contact order, so nobody spends the first ten minutes of an incident figuring out who to call.
None of this existed before the first test. It only got written down because the first test forced the question.
Before and After, at a Glance
| Before first test | After | |
|---|---|---|
| Documented recovery scope | None | Written approach per system type (VM vs. DB) |
| RTO target | Not set | DB: 1 hour, VM: 10 minutes |
| Actual recovery time observed | N/A | DB: ~1 hour, VM: 5–10 minutes |
| Emergency contacts | Ad hoc | Defined escalation order |
| Test cadence | Irregular | Twice a year, formally required |
What the Audit Changed
The audit didn’t just check whether we’d tested. It turned twice-a-year testing from something we happened to do into something we’re now required to document and produce evidence for, going forward. That’s a different kind of pressure than “we should probably test this at some point” — it’s a fixed date on the calendar with a paper trail attached.
Lessons
A disaster recovery plan that’s never been tested is a guess wearing a plan’s clothing. Snapshots existing “somewhere” isn’t the same as having a documented path to actually use them. And the real value of a first test isn’t proving everything works — it’s finding the gaps while nothing is actually on fire.
If your DR plan hasn’t been tested end to end, the technology is probably fine. The documentation is where I’d look first.
Related reading: This wasn’t the first time our backup setup got tested under pressure — see The Day Our Backup Server Went Down for how we ended up here in the first place.
Frequently Asked Questions
Do you need a documented recovery runbook if you already have backups?
Yes. Having backups tells you recovery is technically possible; it doesn’t tell you what “recovered” means for a given system, which recovery point to use, or who makes that call under time pressure. Our first test worked from a technology standpoint and still caused confusion, because none of that was written down.
How often should a disaster recovery plan actually be tested?
Twice a year worked for us — often enough to catch changes in the environment, infrequent enough to be realistic to schedule around a shutdown window. What matters more than the exact frequency is that it’s on a fixed schedule and produces something documented, rather than being an occasional best-effort exercise.
Related reading:
- This wasn’t the first time our backup setup got tested under pressure — see The Day Our Backup Server Went Down for how we ended up here in the first place.
- One specific restore from this same testing cycle turned into its own story — see The Backup Worked. The BMR Didn’t Boot. What We Learned for the physical-to-virtual restore that didn’t go as planned.
- One connectivity issue from this same testing cycle turned into its own story — see We Fixed the Firewall Rule. AWS Authentication Still Failed. for the AWS S3 access problem that didn’t resolve the way I expected.
- Regular testing is exactly what catches the gaps I wrote about in three backup mistakes I’ve seen more than once.