Optimal number of partition for kafka topic on 5 brokers with replication factor=3 in 1 cluster

9
votes

What is optimal no. of partition for a topic that has 5 brokers and replication factor=3? Total there are only 6 topics in this cluster.

Machine configuration that each broker is running as follows.

Memory=16gb
Porcessor= Octacore, Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz

Any challenges other than open file for 1000 partitions per topic??

1
apache-kafka
This blog post should help: confluent.io/blog/… – Matthias J. Sax
Thanks Mathias, this is helpful. As per the article, i should be good for 1500 partitions. "As a rule of thumb, if you care about latency, it’s probably a good idea to limit the number of partitions per broker to 100 x b x r, where b is the number of brokers in a Kafka cluster and r is the replication factor" – Faisal Ahmed Siddiqui

1 Answers

5
votes

Starting since Kafka 1.1.0 it's recommended to have up to 4'000 partitions per broker and up to 200'000 partitions per cluster. Read more here.

Pls, note that even if you can create a large number of partitions it doesn't mean that your hardware will be able to support it. Make sure you scale your cluster hardware(CPU, RAM, HDD etc) accordingly to your load.