kafka Brokers Leader Skewed

Question

I have 10 node kafka cluster .both kafka broker and zookeeper are running on each node. Recently we have added 3 new nodes 8 ,9 and 10 and yesterday 2 node was down (2 and 4). I have a topic with 60 partitions and 3 replication. In kafka manager, Brokers Skewed % is showing as 50 and Brokers Leader Skewed % as 70. I have manually reassigned the partition from UI and Brokers Skewed % is 0 now but it didn't change Brokers Leader Skewed %. I also ran the command:

$ kafka-preferred-replica-election.sh  --zookeeper localhost:2181 --path-to-json-file test.json

Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Created preferred replica election path with ...
Successfully started preferred replica election for partitions Set(..

but it didn't change anything. I can see on UI that brokers 8 and 10 have no leader. How can I rebalance leaders across all brokers evenly? I read that cluster rolling restart of all broker can solve it but I can't (in normal case) restart my production Kafka cluster.

kafka version: 2.3.0
zookeeper version: 3.4.12

added kafka manager screenshot and highlighted the issue with the red circle.

I will appreciate any help.

Answer 1

I was able to solve this issue. if you will see the screenshots carefully, you will notice that none of the preferred replica (first entry of Replicas column) is assigned to broker 8 or 10 and only parition 37 has broker 9 as preferred replica so Preferred Replica Election will not work.

I have shuffled preferred replicas using Manual Partition Assignments (moving broker 8,9 and 10 to replica 0 in 17 paritions (6*3 -2)) option and then running Reassign Partitions and then running Preferred Replica Election.

I hope it will behelpful to others.

Note: 8,9 and 10 node was added recently.