I had just finished renaming a Proxmox node. Every guest was still running, the cluster was quorate, the web UI certificate had been regenerated under the new name. I ran the last check, pinged my router from the host, got four clean replies, and went to open the web UI from my laptop.
Nothing. No certificate warning, no timeout page, just a connection that hung and eventually gave up.
The containers on that host were all still reachable. Nextcloud loaded fine, so did the password vault and the dashboards. Only the host underneath them had gone dark, and only from the outside.
The symptom is the confusing part
Outbound worked perfectly. From the host I could ping anything on the network, resolve DNS, reach the internet, SSH out to another machine. From anywhere else, the host was a black hole. SSH hung, the web UI hung, ping got no reply.
That asymmetry is what sends you looking in the wrong place. Networking problems are usually symmetric. A bad route, a dead link, a wrong VLAN tag, a bridge in the wrong state, all of those break traffic in both directions. So I checked every one of them:
ip -br addr # correct addresses on the right interfaces
ip route # default route present and correct
bridge link # bridge ports up, VLANs correct
ping 192.0.2.1 # replies fineEverything healthy. Meanwhile a tcpdump on the host showed inbound SYN packets arriving from my laptop. They were reaching the machine. Nothing was going back.
At that point the answer is almost always a firewall, and I still nearly missed it, because I had not touched the firewall. I had renamed a host.
The cause
pve-firewall statusStatus: enabled/running (pending changes)That parenthetical is the whole story.
Proxmox keeps firewall configuration in two layers. There is the on-disk config, a cluster-wide cluster.fw plus a per-node host.fw living under /etc/pve/nodes/<hostname>/. And there are the actual running kernel rules, which the pve-firewall daemon generates from that config and pushes into iptables.
Rename the host and the per-node config moves to a new path along with the rest of the node's state. The daemon notices that what is on disk no longer matches what it pushed into the kernel, and it flags the drift as pending changes. What it does not do is apply them. The kernel keeps enforcing the rules it was given before the rename, and those rules scope the management access list to a node name that no longer exists.
Outbound stayed up because the default outbound policy is accept and does not depend on per-node scoping. Inbound died because the accept rule that should have matched my laptop was keyed to the old name.
The guests kept working for a completely different reason: they reach the network through the bridge, not through the host's own IP stack, so the host firewall's input chain was never in their path. That is why the box looked half alive. Half of it was.
The fix
pve-firewall restartUnder a second. Reachability came back before I had finished reading the output.
Note that it is restart, not reload. There is no reload subcommand on Proxmox 8, which is a small thing that will waste a minute if you are guessing at syntax while locked out of a box.
And you want to be on the console or an out-of-band path when you run it, because if you are somehow still connected over SSH through a rule that happens to be working, restarting the firewall is the exact operation that could drop you.
Why this bites so hard
The failure has three properties that make it worse than an ordinary outage.
It is silent. Nothing logs an error. pve-firewall status reports itself as enabled and running, which is true, and buries the important word in brackets after it. If you check services with a script that greps for running, this passes.
It is asymmetric, so it does not look like a firewall. Your instinct on "I can get out but not in" is to look at routing, NAT and the upstream, in roughly that order. A host firewall that lets everything out is not where most people start.
And it is delayed. The rename appears to succeed. Every check you run on the host itself comes back clean, because every check you run on the host itself is outbound. You only discover the problem when you go back to your own machine, which is usually after you have declared the work finished and closed the console.
What I changed
The rename procedure in my own runbook now ends with pve-firewall restart as a numbered step, not a footnote, followed by a verification that has to be run from a different machine:
# on the renamed node
pve-firewall status # must not say "pending changes"# from any OTHER host on the network ping -c 2 <node-ip> curl -sk -o /dev/null -w '%{http_code}\n' https://<node-ip>:8006/ ```
The rule I actually took from it is broader than the command. Any procedure that changes a machine's identity should end with a check performed from somewhere that is not that machine. Everything I ran on the host told me the host was fine. The host was not fine. It just had no way to tell me, because telling me was the thing that was broken.
A note on the addresses
Addresses in this post use the ranges reserved for documentation under RFC 5737. The behaviour is real; the specific addresses are stand-ins so nothing here maps to a live target.
Resources
- Proxmox VE firewall documentation - the config layers and how they are applied
- Renaming a live Proxmox node - the procedure this gotcha belongs to
- More homelab write-ups: iamkay.eu/blog
