When a cloud recovery test starts failing, I usually check our own firewall rule first. That’s how it’s gone the last few times. This time it wasn’t that simple.
The Setup
The cloud recovery test was requested in mid-July, but another issue took priority. We didn’t actually start testing until mid-August. The goal was simple: on top of the on-prem recovery strategy we already had, could we also recover from the cloud if we needed to? Cloud Recovery Vault held the backup data — the test was whether a real recovery from it actually worked.
Where It Broke
August 11. The connection to AWS S3 wasn’t working. I went straight to the firewall rules — a whitelist-based setup, so nothing gets out unless it’s explicitly allowed — and found a typo in one of the allowed URL entries. I’d sent the correct URL to the colleague who manages our firewall changes, and the typo crept in somewhere during entry. An easy fix, and I expected that to be the end of it.
It wasn’t. Traffic was passing through the firewall fine now, but authentication to AWS kept failing anyway. At that point the issue was clearly outside anything on our end, so I opened a case with AWS.
August 19. AWS asked for additional logs. I pulled and sent them.
August 20. Authentication started working. We hadn’t changed anything further on our side, so I verified it with a simple backup upload to S3. It completed without issue.
Then a Second Problem Showed Up
August 24. With the connectivity issue behind us, we ran a full backup-and-restore test against a live VM. I’d verified the target by hostname going in, so I trusted that was enough. It wasn’t until the restored VM didn’t have the functionality I was expecting that something felt off. Checking further, the NetBackup server’s local DNS had that hostname mapped to a completely different IP — it had quietly backed up the wrong machine.
August 25. Once the DNS issue was identified, I re-ran the backup and restore against the correct VM. This time, everything checked out.
What I Took Away From It
We never got a definitive explanation from AWS for what changed. The important part for me was that the firewall issue had already been fixed, the logs had been provided, and authentication started working without another change on our side. The part I didn’t expect was that I never got a clear answer for why.
I still check my own config first — that habit isn’t wrong, and the typo was real. But this one didn’t resolve on our side, and I had to be okay with closing it out without a full explanation. Sometimes the issue really is on the other side, and you may never find out exactly what changed.
The DNS issue on the 24th was its own reminder: hostname alone isn’t a trustworthy way to confirm you’re targeting the right machine if local DNS can quietly drift. A connectivity fix also doesn’t mean the rest of the test goes smoothly — two unrelated problems can show up back to back in the same window.
Related reading: This connectivity issue turned up during the same cycle covered in Why Every Disaster Recovery Plan Needs Regular Testing. The VM mix-up on the 24th traces back to the same kind of issue in my DNS troubleshooting workflow.
Frequently Asked Questions
What should you check first when a cloud recovery test fails?
Your own network path and firewall rules first — that’s usually where the fixable issues are. If traffic is confirmed passing and authentication still fails, the cause may be outside your environment entirely.
What do you do when a vendor-side issue clears without a confirmed root cause?
Document the timeline — what you changed, what the vendor requested, and when the behavior changed. You may not get a full explanation, but the timeline still matters if the same symptom shows up again.