leader election in zookeeper and Kafka

6
votes

a question troubled me a long time. I know the leader election exists in zookeeper. Suppose we have a Kafka cluster connected with Zookeeper ensemble .

So my understanding is like, there is a zookeeper leader in Ensemble, and there are some "partition leaders" in Kafka cluster as well. And Kafka partition leader are elected through Zookeeper API. Is it right ?

Another question, Is master/worker basically same as leader/follower ?

2
apache-kafkaapache-zookeeper

2 Answers

6
votes
  1. Zookeeper leader election has nothing to to with Kafka partition leader election, vice versa.
  2. Kafka partition leader elect by Controller, Controller is the Broker who registers first in Zookeeper's path /Controller/2,2 is broker.id. When a broker crash down, the leader partition (or leader replica) in this broker will be switched by Controller and Controller will choose one replica in ISR be the partition leader. If this partition offline (all replica not available), this partition cannot be used for write or read. If you set unclean.leader.election.enable=true, then the first available replica will be elected as partition leader, no matter it's in ISR or not. So some messages maybe lost.
  3. Kafka and Zookeeper use leader/follower, not master/worker
2
votes

Partition leader election in Kafka is not done through zookeeper API, actually, the first replica in alive assign replicas will be elected as the leader for each partition.

As for the master/worker things, as I know, many open sources have already replaced the old terminology with leader/replica or primary/replica, so basically, they are the same.