what's the difference between kafka-preferred-replica-election.sh and auto.leader.rebalance.enable?

Question

After restart brokers, I found all partitions in one topic's leader were all in broker 3, and I have set imbalance.check.interval.seconds 300, auto.leader.rebalance.enable True but after 300 seconds nothing happened.

So I used bin/kafka-preferred-replica-election.sh, and got what I expected, the leaders were balanced into brokers 1, 2, 3.

I wonder why auto rebalance not happened? What's the difference between kafka-preferred-replica-election.sh and auto.leader.rebalance.enable?

the controller log:

[2019-08-14 09:31:33,454] INFO [Controller id=3] Processing automatic preferred replica leader election (kafka.controller.KafkaController) [2019-08-14 09:31:33,454] TRACE [Controller id=3] Checking need to trigger auto leader balancing (kafka.controller.KafkaController) [2019-08-14 09:31:33,455] DEBUG [Controller id=3] Preferred replicas by broker Map(2 -> Map(__consumer_offsets-22 -> Vector(2), __consumer_offsets-4 -> Vector(2), __consumer_offsets-7 -> Vector(2), __consumer_offsets-46 -> Vector(2), __consumer_offsets-25 -> Vector(2), __consumer_offsets-49 -> Vector(2), __consumer_offsets-16 -> Vector(2), __consumer_offsets-28 -> Vector(2), __consumer_offsets-31 -> Vector(2), __consumer_offsets-37 -> Vector(2), __consumer_offsets-19 -> Vector(2), __consumer_offsets-13 -> Vector(2), fourth_topic-1 -> Vector(2, 3, 1), __consumer_offsets-43 -> Vector(2), __consumer_offsets-1 -> Vector(2), __consumer_offsets-34 -> Vector(2), __consumer_offsets-10 -> Vector(2), __consumer_offsets-40 -> Vector(2)), 1 -> Map(__consumer_offsets-30 -> Vector(1), __consumer_offsets-21 -> Vector(1), __consumer_offsets-27 -> Vector(1), __consumer_offsets-9 -> Vector(1), __consumer_offsets-33 -> Vector(1), __consumer_offsets-36 -> Vector(1), __consumer_offsets-42 -> Vector(1), __consumer_offsets-3 -> Vector(1), __consumer_offsets-18 -> Vector(1), __consumer_offsets-15 -> Vector(1), __consumer_offsets-24 -> Vector(1), __consumer_offsets-48 -> Vector(1), __consumer_offsets-6 -> Vector(1), fourth_topic-0 -> Vector(1, 2, 3), __consumer_offsets-0 -> Vector(1), __consumer_offsets-39 -> Vector(1), __consumer_offsets-12 -> Vector(1), __consumer_offsets-45 -> Vector(1)), 3 -> Map(__consumer_offsets-8 -> Vector(3), __consumer_offsets-35 -> Vector(3), __consumer_offsets-41 -> Vector(3), __consumer_offsets-23 -> Vector(3), __consumer_offsets-47 -> Vector(3), fourth_topic-2 -> Vector(3, 1, 2), __consumer_offsets-38 -> Vector(3), __consumer_offsets-17 -> Vector(3), __consumer_offsets-11 -> Vector(3), __consumer_offsets-2 -> Vector(3), __consumer_offsets-14 -> Vector(3), __consumer_offsets-20 -> Vector(3), __consumer_offsets-44 -> Vector(3), __consumer_offsets-5 -> Vector(3), __consumer_offsets-26 -> Vector(3), __consumer_offsets-29 -> Vector(3), __consumer_offsets-32 -> Vector(3))) (kafka.controller.KafkaController) [2019-08-14 09:31:33,455] DEBUG [Controller id=3] Topics not in preferred replica for broker 2 Map(fourth_topic-1 -> Vector(2, 3, 1)) (kafka.controller.KafkaController) [2019-08-14 09:31:33,455] DEBUG [Controller id=3] Topics not in preferred replica for broker 2 Map(fourth_topic-1 -> Vector(2, 3, 1)) (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] TRACE [Controller id=3] Leader imbalance ratio for broker 2 is 0.05555555555555555 (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] DEBUG [Controller id=3] Topics not in preferred replica for broker 1 Map(fourth_topic-0 -> Vector(1, 2, 3)) (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] TRACE [Controller id=3] Leader imbalance ratio for broker 1 is 0.05555555555555555 (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] DEBUG [Controller id=3] Topics not in preferred replica for broker 3 Map() (kafka.controller.KafkaController) [2019-08-14 09:31:33,456] TRACE [Controller id=3] Leader imbalance ratio for broker 3 is 0.0 (kafka.controller.KafkaController)

but leaders of fourth_topic's partitions are still all in broker 3:

bin/kafka-topics.sh --zookeeper 10.xx.xx.xx:2181/kafka  --topic fourth_topic --describe
Topic:fourth_topic      PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: fourth_topic     Partition: 0    Leader: 3       Replicas: 1,2,3 Isr: 3,1,2
        Topic: fourth_topic     Partition: 1    Leader: 3       Replicas: 2,3,1 Isr: 3,1,2
        Topic: fourth_topic     Partition: 2    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2

Oh I see,since the imbalance ratio is not high enough,thank you — wayuki

Mickael Maison Mickael Maison · Accepted Answer · 2019-08-14T09:09:54

When running kafka-preferred-replica-election.sh, it forces the election of the preferred replica for all partitions.

On the other hand, when you set auto.leader.rebalance.enable to true, the Controller will regularly check the imbalance (every leader.imbalance.check.interval.seconds). However, to avoid unnecessary load on the cluster, leaders are only automatically rebalanced if the imbalance ratio is above leader.imbalance.per.broker.percentage which defaults to 10%.

You can see the current imbalance ratio in the controller log (by default: ${kafka.logs.dir}/controller.log).

what's the difference between kafka-preferred-replica-election.sh and auto.leader.rebalance.enable?

1 Answers