Workflow of Queue Messaging / Consumer Group?

Question

Say I have consumer group G1 which has three consumers C1,C2 and C3. All subscribe to Topic 1 which has single partition. Now Producer produces 3 messages on partition1, broker will handover 1 message to each consumer. Right ?

But I got confused after reading below bold text from section Workflow of Queue Messaging / Consumer Group at kafka tutorial

1. A single consumer subscribes to a specific topic, assume "Topic-01" with "Group ID” as “Group-1”.
2. Kafka interacts with the consumer in the same way as Pub-Sub Messaging until new consumer subscribes the same topic, "Topic-01" with the same “Group ID” as “Group-1”.
3. Once the new consumer arrives, Kafka switches its operation to share mode and shares the data between the two consumers. This sharing will go on until the number of consumers reach the number of partition configured for that particular topic.
4. Once the number of consumer exceeds the number of partitions, the new consumer will not receive any further message until any one of the existing consumer unsubscribes. This scenario arises because each consumer in Kafka will be assigned a minimum of one partition and once all the partitions are assigned to the existing consumers, the new consumers will have to wait.

If i go by point 3/4, will only consumer1 will get all messages as number of consumer (which is 3) is greater than no of partition(which is 1) ? Does it mean i need to create 3 partitions in all and group G1 will subscribe with all 3 partitions so that each consumer gets one message each ?

Answer 1

If i go by point 3/4, will only consumer1 will get all messages as number of consumer (which is 3) is greater than no of partition(which is 1)

Yes. It could be either one of the three.

Does it mean i need to create 3 partitions in all and group G1 will subscribe with all 3 partitions so that each consumer gets one message each ?

Yes, if your topic has 3 or more partition then all 3 consumers will start consuming from your topic.

Answer 2

Kafka provides two different delivery pattern :

• competing/consumer : when consumers are in the same consumer group receiving from a topic, each partition in the topic is assigned to one and only one consumer. It means that if consumers are more than partitions, the remaining consumers will be idle without receiving messages. In your example, the only one partition will be assigned to only one consumer which will be the only one to receive messages; the other consumers will be idle. The number of partitions defines the parallelism in terms of consumers.

• publish/subscribe: when consumers belong to different consumer groups they will receive the same messages (as it happens in the topic/subacriptions of a message broker)

Answer 3

Let us check the Proofed actual workflow of this system :

1. Producers send message to a topic in a regular interval.

2. Kafka stores all messages in the partitions configured for that particular topic similar to the earlier scenario.

3. A single consumer subscribes to a specific topic, assume "Topic-01" with Group ID as "Group-1".

4. Kafka interacts with the consumer in the same way as Pub-Sub Messaging until new consumer subscribes the same topic, "Topic-01" with the same Group ID as "Group-1".

5. Once the new consumer arrives, Kafka switches its operation to share mode and shares the data between the two consumers. This sharing will go on until the number of con-sumers reach the number of partition configured for that particular topic.

6. Once the number of consumer exceeds the number of partitions, the new consumer will not receive any further message until any one of the existing consumer unsubscribes. This scenario arises because each consumer in Kafka will be assigned a minimum of one partition and once all the partitions are assigned to the existing consumers, the new consumers will have to wait.

7. This feature is also called as Consumer Group. In the same way, Kafka will provide the best of both the systems in a very simple and efficient manner.

(I don't want to take any credit please give his credit to Tutorialspoint : https://www.tutorialspoint.com/apache_kafka/apache_kafka_workflow.htm)