Kafka Cluster - issue with one broker not being utilized

0
votes

I am having Kafka cluster with 3 brokers and 3 zookeeper node running. we have added 4th broker recently. When we bring it as new cluster, few partitions got stored in the 4th broker as expected. Replication factor for all topics is 3 and has each topic has 10 partitions.

Later, Whenever we bring down whole kafka cluster for maintenance activity and bring it back, all topic partitions is getting stored in first 3 brokers and no partition is getting stored in 4th broker. (Note: Due to bug, we had to use new log directory every time kafka is brought up, pretty much like a new cluster)

I can see that all 4 brokers is available in zookeeper (when i do ls /brokers/ids i can see 4 broker ids) but partition is not distributed to 4th broker.

But when i trigger partition reassignment to move few partitions to 4th broker, it worked fine and 4th broker started storing the given partition. Both producer and consumer able to send and fetch data form 4th broker respectively.I cant find the reason why this storage imbalance is happening among kafka brokers. Please share your suggestion.

1
apache-kafka

1 Answers

0
votes

When we bring it as new cluster, few partitions got stored in the 4th broker as expected.

This should only be expected when you create new topics or expand partitions of existing ones. Topics do not automatically relocate to new brokers

had to use new log directory every time kafka is brought up

That might explain why data is missing. Unclear what bug you're running into, but this step shouldn't be necessary

when i trigger partition reassignment to move few partitions to 4th broker, it worked fine and 4th broker started storing the given partition. Both producer and consumer able to send and fetch data form 4th broker respectively

This is the correct way to expand a cluster, and sounds like it's working as expected.