how many consumer groups can a kafka topic handle?

Question

suppose I have a kafka topic with say about 10 partitions, I understand that every consumer group should have 10 consumers reading from the topic at any given time to achieve maximum paralellism.

However, I wanted to know if there is any direct rule also for the number of consumer groups a topic can handle at any given point of time. (I was asked this in an interview recently). According to my best knowledge, it depends on the configuration of the broker so as to which how many connections it can handle at any given point of time.

However, just wanted to know how many maximum consumer groups (each with 10 consumers) can be scaled at a given point of time?

Answer 1

In Kafka, there is no explicit limit on the number of consumer groups that can be instantiated for a particular topic. However, you should be aware that the more the consumer groups, the bigger the impact on network utilisation.

Answer 2

As it was said above, up to few thousands, should be okay. For those who will land here (like me) wondering about many thousands of connections (e.g connecting IoT devices directly to kafka), it seems that kafka wasn't designed for that, at least according to this blog.

Answer 3

Conceptually you can think of a consumer group as being a single logical subscriber that happens to be made up of multiple processes. As a multi-subscriber system, Kafka naturally supports having any number of consumer groups for a given topic without duplicating data (additional consumers are actually quite cheap).

As given in the API docs for Kafka 0.9 , Kafka can support any number of consumer groups for given topic.

Link : http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html