There is a category of claim that infrastructure marketing loves: the workload just moves. Node dies, service continues, nobody notices. I wanted to know what that actually looks like on hardware I own, so I picked a production container, powered off the node it was running on, and started a stopwatch.
Ninety seconds later the container was up and serving on the other node. No commands from me after the poweroff. Most of those ninety seconds went somewhere specific, and almost none of the work that made them possible happened that afternoon.
The timeline
At T+0 I issued a clean poweroff on the active node. The container went down with it, and the survivors noticed the silence.
Around T+30, the cluster finished deciding who was in charge. Cluster managers run their own election: the resource manager that decides where high-availability workloads run has a master, and if the master was on the dead node, a new one steps up. Half a minute of that window is bureaucracy, and it is bureaucracy that prevents two nodes both deciding to start the same workload, which would be far worse than waiting.
Around T+90, the container was booted and serving on the surviving node, started from a local copy of its disk that replication had delivered minutes earlier.
The three boring pieces underneath
Each second in that timeline was bought in advance by a specific, unexciting decision.
Replication was already flowing. Every few minutes, an incremental snapshot of the container's disk ships to the partner node. Incremental is the operative word: after the first full send, each cycle moves only changed blocks, which for a small service is seconds of transfer. When the node died, the survivor was holding a disk image at most a few minutes stale. This is the piece people skip, and without it "failover" means "restore from last night's backup," which is a different sport. It stands on the identical-pool-names convention I wrote about last week; replication lands cleanly because the destination looks exactly like the source.
The workload was allowed to fail over to a prepared node only. High-availability groups have a setting that restricts a workload to nodes you list. That reads like a limitation. It is actually the safety: the listed nodes are the ones receiving replication. Without the restriction, the cluster is free to restart the container on a node that has no copy of its disk, and the recovery becomes a hostage negotiation. Restricting placement means the automatic decision is always a correct decision.
The container was small enough to boot fast. Sixty of the ninety seconds are just a Linux container starting. A lean single-purpose container boots in a minute; a bloated one takes five. Failover time is mostly boot time, so every service you keep small is failover you keep fast.
The honest fine print
The replicated copy is minutes stale, so a real crash loses whatever changed inside the replication window. For my workloads, stateless services and things whose state lives in databases elsewhere, that window is acceptable. For a primary database it would not be, and I do not run one this way.
One workaround worth recording: a container originally created on a different storage type cannot be live-migrated onto the replicated ZFS layout. The path that works is backup and restore: take a backup, restore it targeting the right storage, and let replication take over from there. Ten minutes of one-time surgery per guest, and only because those guests predated the storage design.
And the test itself matters more than the setup. Untested failover is a rumor. The first deliberate poweroff is genuinely uncomfortable, which is exactly the argument for doing it on a Tuesday afternoon you chose rather than a night you did not.
Ninety seconds is not luck. It is a naming convention, a replication schedule, a placement restriction, and a small container, each doing its boring job at the same time.
