One RHEL 8.7 server couldn’t establish a Red Hat subscription connection, while a comparable server in the same environment worked normally.
Fixing the first network issue didn’t solve it. It only changed the symptom: no TCP, then TCP without TLS, then a working subscription connection with dnf makecache still failing. This is the incident in that order.
Stage 1: The Server Couldn’t Open TCP
Only this server was affected. I tested against subscription.rhsm.redhat.com, the endpoint the subscription client talks to:
curl: Network is unreachable
nc: Connection timed outThe two tools reported different errors, but both meant the same thing: no TCP connection at all. Since a comparable server worked, I looked at the network path instead of Red Hat or the host.
The first check was the firewall between the corporate network and the internet. A rule for this traffic already existed, so I updated it instead of creating a new one.
The server’s source IP was also missing from the outbound SNAT configuration. After I added the private-to-public mapping, the connection progressed past the point where it had previously failed. The rule update and the SNAT change were both done within about two hours, and TCP then connected.
Stage 2: TCP Worked, TLS Didn’t
The next failure was in the TLS handshake. I tested with OpenSSL against the same endpoint from both servers:
| Failing server | Healthy server | |
|---|---|---|
| DNS | OK | OK |
| TCP/443 | OK | OK |
| Client Hello | Sent | Sent |
| Server Hello | Not received | Received |
| Certificate | None | Received |
| Firewall application label | incomplete | ssl |
The failing server sent a Client Hello and never got a Server Hello back. The firewall classified the healthy session as ssl and the failing one as incomplete.
I also compared OpenSSL and the RHEL crypto policy between the two servers. During the incident the output differed, but only because the failing server never received a certificate. After the fix, the same test returned identical results on both. That comparison let me stop looking at OpenSSL and the crypto policy on the host. The difference was somewhere in the network path.
The remaining change was on the internal firewall, between the production network and the corporate network. It already had a rule for this traffic, but the rule didn’t include an object for subscription.rhsm.redhat.com. In this environment, TCP traffic got through the internal firewall without that object, and the original failure came from the external side. The TLS session completed only after the object was present on both the internal and the external firewall.
This change crossed team boundaries. I flagged both the server owner and the security owner, but ownership wasn’t clear late in the day, and nobody picked it up that evening. The business raised the issue the next morning, and it took about another hour to add the object to the existing rule. After that, the subscription connection went through.
Stage 3: Subscription Worked, dnf Didn’t
Subscription working felt like the finish line, so I ran dnf makecache to confirm the server could actually pull content. It failed.
Red Hat’s subscription service and its content delivery use different hostnames. cdn.redhat.com was missing from the same internal firewall rule. I added it to the existing rule during that final check, and dnf makecache then completed.
What I Took From This
- Follow the changing symptom. No TCP, then TCP without TLS, then a working subscription connection with a failing dnf. Each change moved the failure one layer farther along the connection path.
- Compare with a known-good server. Side-by-side OpenSSL output and the firewall’s
sslversusincompletelabel pointed at the path, not the host. - Validate the actual workload. The subscription connection succeeding didn’t prove package retrieval worked.
dnf makecachedid. - Assign clear ownership for cross-team network changes. The remaining action had no clear owner across the teams involved, and that is what held it overnight. Make ownership explicit before the troubleshooting window ends.
Related reading: We Fixed the Firewall Rule. AWS Authentication Still Failed., another case where the first firewall fix didn’t restore full function.
Frequently Asked Questions
If only one server can’t reach an external service, where should I look first?
Start by comparing the failing host with a known-good host in the same environment. Then map the actual network path for that environment. In mine, that meant checking the external firewall rule, the SNAT configuration, and the internal firewall separately.
Why did TCP work while TLS still failed in this incident?
In this case, establishing TCP only proved that a socket could be opened to port 443. It did not prove that a complete TLS session could traverse every policy point in the path. The failing session sent a Client Hello but never received a Server Hello, while the firewall continued to classify the flow as incomplete.
Why did dnf makecache fail after the Red Hat subscription connection worked?
In this incident, the subscription endpoint and the content delivery host were different hostnames, and the firewall rule covered only the first. After cdn.redhat.com was added to the rule, dnf makecache completed.