I Thought VMware Snapshots Were My Recovery Plan. They Were Only Part of It.

This started as a simple question I thought I already had the answer to: VMware snapshot vs BMR. For a long time, my answer was easy — snapshots covered the everyday stuff, a bad patch or a config change gone wrong, roll back and move on. Bare Metal Restore was the deep end, the thing you’d only reach for if a server was well and truly gone. As long as both existed somewhere in the environment, I considered recovery handled.

I eventually learned that wasn’t the right question.

Where That Assumption Actually Broke

The turning point came when I inherited a database environment that didn’t treat VMware snapshots as the primary safety net before patching at all. OS-level recovery relied on NetBackup and BMR. Database recovery relied on storage-level snapshots at the Hitachi array. VMware snapshots were technically present in the environment, but they weren’t the recovery strategy — they were almost incidental to it.

That was the first time I’d worked somewhere that had already made the decision I’d never really questioned. It didn’t change my thinking immediately. It just planted the idea that “VMware snapshot vs BMR” wasn’t really the right framing — recovery could be built entirely around other layers, and snapshots were just one option among several.

Three separate incidents, none related to each other, are what actually forced that idea into something concrete.

At first, I treated each one as its own isolated problem. Looking back, they all exposed the same gap: I’d been trusting each layer of recovery because I’d never had a reason not to, not because I’d ever confirmed it would actually hold up.

The Restore That Wouldn’t Boot

The first incident was direct: a restore that reported success right up until the server tried to boot. The backup had worked — every check NetBackup had along the way came back green. Booting from it was a different story. That was the first time “the backup completed” and “the server is recoverable” stopped meaning the same thing to me.

At the time, I treated it as a one-off — a BMR quirk, something specific to that server’s configuration. It took the next two incidents before I saw it as the first data point in a pattern.

It Wasn’t Just Recovery That Failed

The second incident wasn’t about a restore at all. It was the backup infrastructure itself — a media volume on our backup server filling up and taking the backup service down with it. Nothing about the VMs being protected had changed. The layer meant to protect everything else had a failure mode of its own, one that had nothing to do with snapshots, BMR, or anything I’d been thinking about after the first incident.

The backup jobs themselves weren’t the problem. The platform running them was.

Then the Blind Spot Showed Up Somewhere Else Entirely

By then, I’d already learned that recovery depended on more than backups completing successfully. The third incident showed me that monitoring was another layer I’d been taking for granted just as much.

The third one didn’t touch backup or recovery at all on the surface. A Zabbix proxy went dark for hours, taken down by an unattended OS update that had nothing to do with monitoring, and nothing in the environment raised a flag — because the thing that would normally raise the flag was the thing that had gone quiet.

That’s when the pattern actually landed. A snapshot-based backup that reported success. A backup server that just sat there working, month after month. A monitoring proxy that, by definition, never told me it might fail. None of them lied. They just each had a blind spot I hadn’t gone looking for.

What Changed

I stopped treating “it’s backed up” as the end of the conversation. A snapshot or a backup job completing tells you the write happened — not that the thing on the other end will actually come back to life, or that the infrastructure underneath it will hold, or that you’d even find out if it didn’t.

VMware snapshot vs BMR was never really the question. Both are just mechanisms, and the environment that relied on neither for its database tier was proof that the choice itself isn’t what matters. The real question is whether you’ve actually verified that whichever mechanism you picked does what you assume it does — and how you’d know if it stopped.

Lessons Learned

A successful backup and a working recovery are two different claims, and only testing a restore actually confirms the second one.

Recovery design isn’t one layer. The snapshot, the backup infrastructure, and the monitoring that watches both are three separate places a silent failure can hide.

Related reading:

Frequently Asked Questions

Should I use VMware snapshots or NetBackup BMR for disaster recovery?

They’re not really competing options — they cover different scenarios, and most environments need both, or in some cases neither for a given tier. The bigger question isn’t which one to pick, but whether you’ve actually tested that it recovers a system the way you expect.

Are VMware snapshots the same thing as a backup?

Not really. A snapshot is a point-in-time reference that depends on the original disk still existing — it’s built for quick rollback, not long-term protection. A backup, done properly through something like NetBackup, is an independent copy that doesn’t depend on the source surviving.

Why would a production database server avoid VMware snapshots before patching?

In environments where database consistency matters more than convenience, storage-level snapshots (at the array, not the hypervisor) and application-aware backups can give a cleaner, more reliable recovery point than a VMware snapshot — especially for databases sensitive to I/O overhead or crash-consistent (rather than application-consistent) snapshots.

How do you actually test whether a backup will restore successfully?

The only real test is doing the restore itself — ideally to isolated hardware or an isolated environment, not just checking that the backup job finished without errors.