The Preferred Replica assignment of Kafka

Question

When we create a topic in kafka, broker assigns replicas for partitions in this topic, and the First Replica will be the Preferred Replica which means that kafka cluster will migrate partition leader to Preferred Replica on the basis of 'imbalance rate'.

Consider that with the increasing of the brokers, the partitions Preferred Replicas are always the one assigned when created those topic. So the load balancing is not scalable with the change of the scale of the brokers.

So I would like to ask,there is any strategy that can modify the assignment of the Preferred Replica when brokers increase in the current version of kafka.

Answer 1

The Kafka documentation has a section about this: http://kafka.apache.org/documentation/#basic_ops_cluster_expansion

When expanding your cluster, you can use the reassignment tool, kafka-reassign-partitions.sh, to move some of the existing assignments onto the new brokers.

Kafka doesn't provide a way to automate that, but there are third party tools that offer that like https://github.com/linkedin/cruise-control