inconsistent state after zookeeper leader crash?

Question

I'm trying to understand zookeeper's internal.

Suppose a 3-servers zookeeper cluster, the leader server send a proposal(say setdata: foo=1) to two followers and then crashed, but at least one follower record this proposal to its transaction log file. According "Zab paper" says, the other two server can still form a valid quorum and elect a new leader. And the new leader can still propose and commit this proposal(setdata: foo=1).

My question is in this situation, the client think this request is not completed(because of the leader crash and not respond or the client timeout), but in fact it is still success in the zookeeper cluster. Is this an inconsistent?

Answer 1

In fact this is an inconsistent, but it's not a problem.
In zookeeper programmer guide,there is a line:

If a client gets a successful return code, the update will have been applied. On some failures (communication errors, timeouts, etc) the client will not know if the update has applied or not. We take steps to minimize the failures, but the only guarantee is only present with successful return codes. (This is called the monotonicity condition in Paxos.)

This means you know your update succeed when you gets a successful return code, but when you can't know whether it succeed or failed when you don't receive the return code.

But this is not a problem, when your update failed because of leader crash, you can just retry the update operation. This time your update will failed because the the version you specified is behind the actual version number and you will be notified. Then you can call get method to retrive the data to see whether the data equals you specified value.