Apache Kafka partition re-assignment upon full cluster restart reg

0
votes

I have a Kafka cluster of 3 nodes. I want to add 2 more nodes to the cluster to make it 5 node cluster. After adding the nodes, I will do a FULL CLUSTER RESTART. i.e., shutdown all kafka & zookeeper services at once and then start all the services in 5 nodes.

I want to know 1. Does the FULL CLUSTER RESTART distribute the topic partitions evenly across 5 nodes? Or need to run the partition reassignment tool kafka-reassign-partitions.sh ? 2. Doesn't Kafka have internal mechanism to automatically reassign the partitions unless running kafka-reassign-partitions.sh

Please clarify. Thank you.

1
apache-kafka
When adding a new node to the cluster, it is not necessary to restart the existing nodes of the kafka and zookeeper. Only new topics will be placed evenly across all brokers, old ones need to be balanced by hand – Andrey

1 Answers

0
votes

No, it will not rebalance the partitions across nodes. You will need to do this manually. Kafka will rebalance consumers when a new consumer joins a consumer group but that is something totally different.

If you are running the full Confluent stack you can use the confluent-rebalance tool to do it.