Two node ZooKeeper ensemble

Question

We have a setup with exactly two data centers, we have exactly one machine in each data centre and we wish to have a ZooKeeper node deployed to both these machines, and for them to be clustered. We are unable to add any new machines to the cluster.

Currently we have two ZooKeeper nodes A and B, one on machine A and one on B. We must be able to tolerate the failure of a single machine and the other ZooKeeper instance must remain operable.

I understand from the ZooKeeper documentation that a minimum of three nodes are advised - as the coordination mechanism requires a majority of nodes to be alive to continue (in that case two).

My question is - is it possible to get a highly available ZooKeeper deployment that consists of only two nodes?

Answer 1

No. In case of Zookeeper 2 nodes is not enough. Even if one of your nodes went out, the remaining one couldn't tell if it was down or just partitioned. This would cause inconsistency, a split brain scenario, because both of them would continue serving their clients.

I am not sure if Zookeeper would even form a cluster or just run in standalone mode. Check out this answer: Zookeeper running on two nodes

In your case, with only 2 machines, you might consider putting a second Zookeeper instance on one of the machines and making a 3 node cluster.

This wouldn't do you much good in terms of fault tolerance. It would be similar to just running a standalone ZK on the machine that has the second instance. It might speed up read request, or make it easier for you to migrate to a proper cluster setup once you grow to 3 or more machines. It will also add to your operational complexity and write speed.

You might also look into hierarchical quorums and observers which might be useful for clusters spanning multiple datacenters.