Automatic NameNode failover

Now that we have introduced ZooKeeper, we can show how it is used to enable automatic NameNode failover.

Automatic NameNode failover introduces two new components to the system, a ZooKeeper quorum, and the ZooKeeper Failover Controller (ZKFC), which runs on each NameNode host. The ZKFC creates an ephemeral ZNode in ZooKeeper and holds this ZNode for as long as it detects the local NameNode to be alive and functioning correctly. It determines this by continuously sending simple health-check requests to the NameNode, and if the NameNode fails to respond correctly over a short period of time the ZKFC will assume the NameNode has failed. If a NameNode machine crashes or otherwise fails, the ZKFC session in ZooKeeper will be closed and the ephemeral ZNode will also be automatically removed.

The ZKFC processes are also monitoring the ZNodes of the other NameNodes in the cluster. If the ZKFC on the standby NameNode host sees the existing master ZNode disappear, it will assume the master has failed and will attempt a failover. It does this by trying to acquire the lock for the NameNode (through the protocol described in the ZooKeeper section) and if successful will initiate a failover through the same fencing/promotion mechanism described earlier.