Does Kafka Preferred leader gets updated ever, even if the brokers are all up?

Question

I am trying to make a kafka monitoring service, which needs to have at least one partition of a topic on each broker in the cluster. I assigned the partitions initially. I have set the replication factor to 1, and min.insync.replicas is also 1, and I am producing with a sync producer with acks = all.

The partition-broker assignment looks like this initially

Partition 0 : Broker 0 (Leader) , Broker 0 (Preferred Leader) 
Partition 1 : Broker 1 (Leader) , Broker 1 (Preferred Leader) 
Partition 2 : Broker 2 (Leader) , Broker 2 (Preferred Leader)

When I try to kill a broker (ex. broker 2), the partition whose leader was that broker, should and is reporting "No Leader" error. Now since I have 1 replication and only the leader is the in sync replica, kafka would not re-elect any other broker as leader, which is expected and until here everything is fine.

At this point the partition-broker assignment looks like,

Partition 0 : Broker 0 (Leader) , Broker 0 (Preferred Leader)  
Partition 1 : Broker 1 (Leader) , Broker 1 (Preferred Leader)  
Partition 2 : -1                , Broker 2 (Preferred Leader)

QUESTION: But when that broker comes back up, both the leader and the preferred leader of the partitions gets updated to some other broker. I don't understand this, and isn't expected. So, am I missing any configs, or has anyone ever faced this issue?

The partition-broker assignment goes like this. Whyy??

Partition 0 : Broker 0 (Leader) , Broker 0 (Preferred Leader) 
Partition 1 : Broker 0 (Leader) , Broker 0 (Preferred Leader) 
Partition 2 : Broker 0 (Leader) , Broker 0 (Preferred Leader)

btw, If you set replication factor to 1, then acks=all is the same as acks=1 — OneCricketeer
And I don't really understand the question. How do you get this output? Leaders cannot be assigned to brokers where the data doesn't exist... Seems like you may have killed the controller broker or Zookeeper — OneCricketeer
I am getting this output on kafka manager. I haven't touched zookeeper or the controller broker, actually I just noticed that, even without the broker getting killed, the preferred and the leader is shifting, i don't understand this behavior. Does kafka internally has any reassignment strategy, if I have very low qps? I produce very low qps, and there's always 0 consumer lag in my setup. — srnand
How are you checking which broker is the controller? That is one of the brokers, and it is responsible for leader management while it communicates with Zookeeper — OneCricketeer

srnand srnand · Accepted Answer · 2020-03-31T01:15:25

There wasn't anything wrong with kafka. I found out it was cruise control self healing, which wasn't working as expected (not excluding topics).

Does Kafka Preferred leader gets updated ever, even if the brokers are all up?

1 Answers