I was mid-way through bringing a new server onto the network when I picked an address for it. The next free one in the range, or so I thought. I said the number out loud while typing it, the way you do, and something about it felt familiar.
It was the password vault's address. It had been the password vault's address for two weeks. I had checked the machines on the network, seen nothing at that number, and moved on, because the vault was a container running on a different host and my check had only ever been going to find things on the host I was standing on.
That one I caught before it did anything. The fix was to move the vault to the next address up and let the new server keep the one I had already configured at the switch: stop the container, edit its address, update the reverse proxy backend and the internal DNS record, start it again. Ten minutes.
A few weeks later I did the same thing again and did not catch it, and that is the interesting half of this story.
The check that does not check
On a Proxmox cluster the obvious way to see what is running is to ask:
pct list
qm listBoth of those tell you about guests on the node you are logged into. They say nothing about the guests on the other nodes. If you have three hosts and you run this on one of them, you are looking at roughly a third of your estate and it does not look partial. It looks like a complete list, because it is a complete list of something.
A ping test is no better, and is worse in one specific way. It tells you whether something answers right now. A container that is stopped, or a machine that is powered off, still holds its claim on that address in its configuration, and will take it back the moment it starts.
So the second time, I provisioned a new container on the address my checks said was free. It was not free. A container on a different node had held it since a build I had left half-finished weeks earlier and had stopped thinking about.
What a duplicate address actually looks like
This is the part worth internalising, because the symptoms do not spell out "duplicate address" at all.
Ping worked from every node, consistently. Two machines both received the address resolution request, both answered it, and whichever reply arrived first won that particular exchange. Ping does not care which one replies, so it looked completely healthy from every angle I tested.
Then the actual service failed, and it failed differently depending on where I tested from.
From most machines, connecting to the application port returned connection refused. Those machines had cached the other container's hardware address, and that container has nothing listening on that port.
From one machine, everything worked perfectly. It happened to have cached the correct hardware address, so it reached the right container every time. That single working case cost me a good twenty minutes, because a service that works from exactly one place looks like a firewall rule or a routing problem, not an addressing problem.
And the reverse proxy, which sat on a node that had cached the wrong address, served an unbroken run of 502s while I stood on the one node where curl was returning 200.
Flushing the address cache does not fix this, which is the tell. The wrong machine is still there, still configured with that address, still answering resolution requests. Clear the cache and it repopulates, sometimes correctly, sometimes not. Anything that gets better on one machine and stays broken on another, and shuffles when you clear a cache, is an addressing conflict until proven otherwise.
The check that actually works
Every guest on a Proxmox cluster has its config file replicated to every node through the cluster filesystem. So the authoritative question is not "what is running here" but "what does the cluster believe it has handed out."
grep -rhE "ip=192\.0\.2\." \
/etc/pve/qemu-server/ /etc/pve/lxc/ \
/etc/pve/nodes/*/qemu-server/ /etc/pve/nodes/*/lxc/ 2>/dev/null \
| grep -oE "192\.0\.2\.[0-9]+" | sort -u -t. -k4 -nThat returns every address claimed by every guest on every node, running or stopped, in numeric order. Gaps in the output are genuinely free. It takes about a second and it is the only check in this post that is actually authoritative.
Ping the candidate afterwards as a belt-and-braces step, by all means. But the config grep is the source of truth, and the ping is not, because a stopped guest is invisible to ping and still owns its number.
The general shape of the mistake
Both times, the failure was the same one: I ran a check whose scope was narrower than the question I was asking.
pct list answers "what is on this node." I was asking "what is on this cluster." Ping answers "what is awake at this address." I was asking "what is assigned to this address." Neither tool was wrong. Both gave complete, accurate answers to questions I had not meant to ask.
That is a much easier failure to walk into than a tool that gives you a wrong answer, because a wrong answer eventually contradicts something. A correctly-scoped answer to the wrong question agrees with everything, right up until it does not.
The habit I have taken from it is to say the question out loud before choosing the command, specifically the scope of it. Not "is this address free" but "is this address free anywhere in the cluster, including on things that are currently switched off." Phrased that way, pct list obviously does not answer it, and the right command is the one you reach for instead.
A note on the addresses
Addresses in this post use the range reserved for documentation under RFC 5737. The incidents and the behaviour are real; the numbers are stand-ins.
Resources
- Proxmox cluster file system) - why every node holds every config
- RFC 826, Address Resolution Protocol - why the first reply wins
- More homelab write-ups: iamkay.eu/blog
