Seven incidents in one year taught me more about being an infrastructure engineer than any certification did. None of them were huge disasters.
A stale VMware snapshot nobody deleted. A backup server that went down from a full media disk. A backup policy that was never created for three new servers. A Zabbix alert threshold that didn’t catch a server going down soon enough. A firewall rule nobody remembered, still blocking traffic on a host nobody had touched in years. A DR test that exposed a documentation problem instead of a technical one. A DNS issue that turned out not to be a DNS issue at all.
On paper, they’re unrelated — different systems, different clients, different root causes. But looking back, I keep seeing the same thing. The environment wasn’t always what we thought it was.
A snapshot that should have been removed was still there. A backup policy that should have existed didn’t. An alert that should have fired wasn’t configured to — and the server was down for a while before anyone noticed. A firewall rule that nobody remembered was still active.
In most of these cases, the technology itself wasn’t broken. It was doing exactly what it had been configured to do. The problem was that the configuration didn’t match what everyone assumed was there.
What I’ve Learned
You can learn VMware. You can learn NetBackup. You can learn Zabbix, DNS, firewalls, storage, whatever happens to be in your environment. That part just takes time.
The harder skill is learning not to assume. Check the snapshot instead of assuming someone already cleaned it up. Check the backup policy instead of assuming it exists because the server shows up in the system. Check the alert instead of assuming you’ll be notified when something breaks. Check the firewall rule instead of assuming it was removed when the project that needed it ended.
Checking Instead of Assuming
I’ve seen the same pattern across very different environments — about twelve years in Germany, some time back in Korea, and now here in the US. It wasn’t so much a national culture thing as it was each company I worked for, but the contrast was still real. In Germany, everything started with a document and ended with a document — if it wasn’t written down, it basically didn’t count as decided. In Korea, there was plenty of documentation sitting around, but nobody really referenced it before doing the work; you just picked things up and got moving. Here in the US, it’s felt more like things start with a conversation and end with a conversation — the doc, if it exists at all, is an afterthought.
None of those approaches were wrong, exactly. But all three left the same kind of gap open: something everyone assumed was documented, or assumed was common knowledge, or assumed had already been said out loud to the right person. The gap between what should be there and what actually is there didn’t change. Only the shape of the assumption did.
These incidents changed how I approach environments I inherit now. I don’t assume something is configured just because the documentation says so. I check. It’s a small habit, but after seven incidents, I trust it a lot more than I used to.
Frequently Asked Questions
What’s the most important skill for an infrastructure engineer?
Technical knowledge matters, but the skill that prevents the most incidents is verification — checking what’s actually configured in an environment rather than trusting documentation, handoff notes, or assumptions carried over from other environments.
Why do infrastructure incidents so often trace back to something other than a technical failure?
Technology generally does exactly what it’s configured to do. The failure usually sits in the space around it — a manual step nobody automated, a policy nobody re-checked after the environment changed, or ownership of a system that changed hands without every assumption changing hands with it.