My kafka version is kafka_2.9.2-0.8.1.1. I have two brokers in the cluster, 4 topics and each topic has 4 partitions.
When I run
sh kafka-topics.sh --describe --zookeeper rhost:2181
for all the topics/partitions, I see broker 1 as Leader.
How can I load balance the leader?
For example, for topic 1 and topic 2 have broker 1 as leader and for topic 3 and topic 4 have broker 2 as leader.
The partitions should be automatically rebalanced, since the default value of the broker configuration parameter auto.leader.rebalance.enable is true. (see documentation)
However, by default this rebalance occurs every 5 minutes, as defined by the leader.imbalance.check.interval.seconds parameter. If you wish this to occur more frequently, you will have to modify this parameter.
You can use the Preferred Replica Leader Election Tool:
sh kafka-preferred-replica-election.sh --zookeeper zklist
This guarantees that the leadership load across the brokers in a cluster is evenly balanced.
I know it is a bit late, maybe you already have the answer, But to balance leaders, first you need to make brokers equally preferred between all partitions, for a broker to be a preferred leader it has two criterion First it needs to be insync replica, Second it has to be the first element on the replicas list (weird IMO), So if you have a small enough number of topics/partitions do that manually it would be easier, otherwise you need to reassign partitions with distributing the first element (preferred replica) among all your brokers, then kick off preferred leader election tool which will make sure that the preferred leader is actually the leader.
