How Kafka broadcast to many Consumer Groups

Question

I am new to Kafka and I will appreciate very much clarification on the next case.

Kafka documentation says in the paragraph "Consumer Position":

"Our topic is divided into a set of totally ordered partitions, each of which is consumed by one consumer at any given time."

Based on statement above if few Consumer Groups subscribed to a topic and Producer will publish message to particular partition within this topic then only one Consumer can pull the message.

The question is how broadcast to many Consumer Groups could happen if only one Consumer can pull particular message?

Answer 1

if there are 10 partitions for a topic and 3 consumer instances (C1,C2,C3 started in that order) all belonging to the same Consumer Group, we can have different consumption models that allow read parallelism as below

Each consumer uses a single stream. In this model, when C1 starts all 10 partitions of the topic are mapped to the same stream and C1 starts consuming from that stream. When C2 starts, Kafka rebalances the partitions between the two streams. So, each stream will be assigned to 5 partitions(depending on the rebalance algorithm it might also be 4 vs 6) and each consumer consumes from its stream. Similarly, when C3 starts, the partitions are again rebalanced between the 3 streams. Note that in this model, when consuming from a stream assigned to more than one partition, the order of messages will be jumbled between partitions.

Each consumer uses more than one stream (say C1 uses 3, C2 uses 3 and C3 uses 4). In this model, when C1 starts, all the 10 partitions are assigned to the 3 streams and C1 can consume from the 3 streams concurrently using multiple threads. When C2 starts, the partitions are rebalanced between the 6 streams and similarly when C3 starts, the partitions are rebalanced between the 10 streams. Each consumer can consume concurrently from multiple streams. Note that the number of streams and partitions here are equal. In case the number of streams exceed the partitions, some streams will not get any messages as they will not be assigned any partitions.

If there is another consumer group, the same process is applied to consumers within that consumer group

Answer 2

Only one consumer in a consumer group can pull the message. But all consumer groups get the messages.

So if you want all your consumers to get the messages, assign them different consumer groups. Each message goes to every consumer group, but within a group, it goes to only one consumer.

Read the Consumer section here.

Answer 3

Generally there are 2 kinds of messaging patterns:

1. Shared queue: All consumers subscribe to one single message queue. Each consumer compete with each other and for each message, only one consumer will get it.
2. Publish-subscribe: Each message is broadcast to all consumers subscribed. So all consumers will get the same message.

Kafka supports both of them at the same time through the concept of consumer group. Consumers in the same group follow the shared queue pattern. Only one consumer in a group can get the message.

Different consumer groups follow the publish-subscribe pattern. For each message, all consumer groups that subscribed to the topic will get a copy of the message.

A useful reference: https://dzone.com/articles/dont-use-apache-kafka-consumer-groups-the-wrong-wa

Answer 4

Good Question.

Take a example i m having a topic called complaint having two partition p1,p2

now i m having two consumer group called group1 having two consumer c1 and c2 and group2 having consume c3

here i route messages from p1 should go to c1 and p2 should go to c2 and i subscribed another consumer called c3 but that is in different group so here copy of the whole message is send to that consumer also