How to configure the time it takes for a kafka cluster to re-elect partition leaders after stopping and restarting a broker?

6
votes

I have the following setup:
3 kafka brokers and a 3 zookeeper ensamble
1 topic with 12 partitions and 3 replicas (each kafka broker is thus the leader of 4 partitions)

I stop one of the brokers - it gets removed from the cluster, leadership of its partitions is moved to the two remaining brokers

I start the broker back - it reappears in the cluster, and eventually the leadership gets rebalanced so each broker is the leader of 4 partitions.

It works OK, except I find the time spent before the rebalancing too long (like minutes). This happens under no load - no messages are sent to the cluster, no messages are consumed.

Kafka version 0.9.0.0, zookeeper 3.4.6

zookeeper tickTime = 2000

kafka zookeeper.connection.timeout.ms = 6000

(basically the default config)

Does anyone know what config parameters in kafka and/or zookeeper influence the time taken for the leader rabalancing ?

1
apache-kafkaapache-zookeeper
is auto.leader.rebalance.enable=true set? – Nautilus
yes it is, - it is true by default and the rebalancing works fine, just takes more time than i would like to.. – Jan Šourek

1 Answers

6
votes

as said in the official documentation http://kafka.apache.org/documentation.html#configuration (More details about broker configuration can be found in the scala class kafka.server.KafkaConfig.) there actually is a leader.imbalance.check.interval.seconds property which defaults to 300 (5 minutes), setting it to 30 seconds does what I need.