It was around half past midnight. I was renaming a Proxmox node, the last real step in a five hour session that had already gone well. Two commands left. I ran them in the order they appeared in my notes.
cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/
rm -rf /etc/pve/nodes/oldname/The copy printed nothing. The delete printed nothing. Both of those are normal outputs for a successful run, so I moved on to the next thing.
The copy had failed. The destination subdirectories did not exist yet, because the daemon that owns that filesystem had created the parent directory but had not finished creating the children. A copy into a path that is not there does not error loudly. It reports that it did nothing, which on a terminal at midnight is indistinguishable from reporting that it did everything.
So the second command deleted the only canonical copy of the configuration for twelve running production services.
What was actually gone
Every guest on that host was defined by a small text file: which disks it mounts, which network bridge it attaches to, how much memory it gets, which VLAN it sits on, what its boot order is. Twelve of those, covering the family file storage, the password vault, the reverse proxy, both DNS resolvers, the monitoring stack, the git server.
Not the data. The data was untouched, sitting on its volumes exactly where it had always been. What I had destroyed was the description of how any of it fits together. Which is arguably worse, because data you can point at. A container with no config is not a container, it is an unlabelled disk image.
Why nothing went down
Here is the part that saved me, and it is worth understanding properly rather than treating it as luck.
A running container or virtual machine does not consult its config file while it runs. The management layer reads that file once, at start, uses it to build the runtime, and then the running process carries on from what it already has in memory. Nothing re-reads the directory during normal operation. The management daemon only rebuilds its picture of the world from those files on a cold start.
So for as long as nothing restarted, every one of those twelve services kept serving traffic as though nothing had happened. Nextcloud carried on syncing, the vault carried on unlocking, and DNS never missed a query. My family noticed nothing, that night or ever.
That is a genuine forgiveness window, and it is a real property of how the system is built rather than a fluke. It is also the most dangerous kind of safety net, because it is invisible. Nothing tells you that you are inside it. If I had rebooted that host, or if the power had cut, I would have come back up to twelve stopped services and no record of how any of them were configured. The window was open. It was not going to announce when it closed.
The recovery
I had taken a full copy of the whole configuration tree to a plain directory on local disk about thirty minutes earlier, as the first step of the rename. Not a snapshot inside the same filesystem I was about to break, an ordinary directory somewhere else that I could read with cat when everything clever had stopped working.
cp -a /etc/pve /root/etc-pve.backup-pre-rename-20260617-001153Recovery was copying the config files back into the correct node directory, once it existed properly, and restarting the management services. Around four minutes of work. All twelve guests reappeared in the interface still running, with the same process IDs they had held before the rename started, because they had never stopped.
That backup was the only reason this is a story about a mistake rather than a story about a rebuild.
What actually went wrong
It is tempting to file this under "be careful with rm -rf" and move on. That is not the lesson, and treating it as the lesson is how you make the same mistake again with a different command.
The real error is that I turned an operation with built-in safety into two operations without it. mv is atomic. It either moves the thing or it does not, and it will not leave you in a state where the source is gone and the destination is empty. I could not use mv here, because the daemon had already created the destination and mv refuses to move onto a non-empty directory. So I reconstructed mv by hand out of cp and rm.
The moment you do that, you have taken on a job the kernel was doing for you: checking that the first half succeeded before running the second half. I did not do that job. I just typed both halves.
The second error is that I read silence as success. cp -av is verbose, that is what the v is for, and it printed no filenames because it copied no files. I saw an absence of errors where I should have seen an absence of output. Those are different things, and at midnight they look identical.
The rule I actually wrote down
Verify the destination before destroying the source. Every destructive command needs a gate in front of it, and the gate has to check for presence, not for the absence of an error.
In practice, for this specific operation:
cp -av /etc/pve/nodes/oldname/lxc/. /etc/pve/nodes/newname/lxc/# the gate src=$(ls /etc/pve/nodes/oldname/lxc/ | wc -l) dst=$(ls /etc/pve/nodes/newname/lxc/ | wc -l) echo "source: $src destination: $dst" [ "$src" -eq "$dst" ] && [ "$src" -gt 0 ] || { echo "MISMATCH, stopping"; exit 1; } ```
Four extra lines. It would have caught this instantly, because the destination count would have been zero and the source count twelve.
There is also a simpler answer that I have adopted since, which is to not run the delete at all. Nothing required me to remove the old directory. It was a handful of small text files taking up a trivial amount of space on a machine with hundreds of gigabytes free. I deleted it because leaving it there felt messy.
That is the part that stays with me. The only genuinely destructive command in a thirty minute procedure was the one that served no purpose. It was not in service of the goal. It was housekeeping. And housekeeping is a bad reason to run rm -rf against the only copy of anything.
What I would tell someone about to do this
Take the backup to somewhere outside the system you are about to modify, and take it first, before you feel like you need it.
Put a verification gate between any copy and any delete that depends on it, and make the gate assert that files are present rather than assert that nothing errored.
Do the destructive work at the start of an evening, not the end. This was a sequencing failure, and sequencing is the first thing that degrades when you are tired and the checklist is nearly finished.
And if a step exists only because the leftovers look untidy, consider not doing it.
A note on the names
Hostnames and paths in this post use documentation placeholders. The incident, the mechanism and the recovery are exactly as they happened.
Resources
- Renaming a live Proxmox node - the procedure this happened inside
- Proxmox pmxcfs documentation) - why the destination gets created for you
- More homelab write-ups: iamkay.eu/blog
