The day I formed my three-node Proxmox cluster, the create went cleanly, the first join went cleanly, and the cluster reported quorate. Every green light a person could ask for. It took a closer look at the config to notice that one node's cluster heartbeat was running over the wrong network entirely, and that nothing anywhere was going to warn me about it.
The one-flag mistake
Forming a Proxmox cluster is two commands. On the first node:
pvecm create clusternameAnd on each joining node:
pvecm add <address-of-first-node>That second command took my instruction about which node to join, and quietly made its own decision about which of the joining node's addresses to register as the cluster link. My nodes are multi-homed: each has an address on the server network, where cluster traffic belongs, and another on a management network that exists for admin access. I pointed pvecm add at the first node's server-network address and assumed the joining node would present its matching interface.
It presented the management interface instead. The flag I did not pass is --link0, which pins which local address the node registers for corosync, the protocol that carries cluster membership and configuration consensus. Without it, the node picks by its own resolution logic, and what it picks is whatever your naming and routing happen to serve up, which on a multi-homed box is a coin toss you did not know was being flipped.
The result: two nodes heartbeating on the server network, one heartbeating from its management address, and a cluster that reports healthy because packets do, in fact, arrive. For now.
Why "it works" was the dangerous part
If the wrong link had failed to connect, I would have caught it in the first minute. Instead everything worked, because the management network happened to route to the others. The problem was latent, and it had teeth:
The management network is not built for cluster traffic. It is the network I take down, re-firewall, and generally treat as a maintenance surface. Corosync is the one protocol that must never flap: if a node loses heartbeat for long enough, the cluster considers it dead, and with high availability enabled, a node that loses quorum will fence itself, a deliberate hard reset to protect shared state. A firewall change on a network I believed was "just admin access" could have rebooted a production node.
That failure would have arrived months later, disconnected from its cause, on an afternoon when I touched something apparently unrelated. The bugs that pass every test and then sit waiting are the worst kind in infrastructure.
The fix: editing the cluster's own consensus file
The corosync config lives in the cluster filesystem, and you fix a wrong ring address by editing it directly, carefully:
cp /etc/pve/corosync.conf /root/corosync.conf.backup
nano /etc/pve/corosync.confInside, each node has a ring0_addr. I corrected the wrong one to the node's server-network address, and, critically, incremented config_version in the totem block. That version bump is how corosync knows this is a newer config to converge on rather than a conflicting one to argue about. Skip it and nodes can reject the change or, worse, split over it.
Then restart corosync on every node, one at a time, watching quorum recover between each:
systemctl restart corosync
pvecm statusTotal fix time, once understood: about ten minutes. The cluster never lost quorum during the rolling restart, and the ring has lived on the correct network since.
What I actually learned
The command that joins a node to a cluster takes a flag that pins which network the cluster's nervous system runs on. On any multi-homed node, that flag is not optional politeness; it is the difference between choosing your topology and being assigned one.
So the join I run now, every time, looks like this:
pvecm add <first-node> --link0 <this-node's-cluster-address>And the check after any join, no matter how green the output: read the config back and confirm every ring0_addr is an address you chose.
grep ring0_addr /etc/pve/corosync.confGreen lights tell you packets arrived. They do not tell you the packets took the road you intended. On the systems that fence themselves when the road washes out, you check the road.
