Kafka Topic And Offset

Question

I have a question,Suppose there is a TOPIC T1 and there are two Consumer C1 and C2 belongs to two different groups and current offset is 0.We know that Kafka maintains the Offset for consumer.So if C1 consume the message and Offset becomes 1 then what will happen if C2 consume the message will it start from 1 offset or it will start consume the message from 0 offset?. Means how offset will be maintains for two different consumer Group?.

Thanks

Answer 1

Kafka maintains the consumer offsets per consumer group, topic and partition, which means, if you have multiple consumers with different consumer groups on the same topic, it will maintain the offset for each group individually. So, in that case, if consumer 1 reads messages, the offset for consumer 2 will not be affected, as long as it belongs to a different consumer group.

Answer 2

As You have mentioned there are two consumer with different group. So In that case C2 will consume from the begining. It means offset will start from 0 in C2 consumer. Offset might be different only If you will use same group in both consumer.

Answer 3

If you have two Kafka consumers with different Group Id they will read partitions without any interference between each other. Meaning both consumers will read the exact same set of messages independently. If you have four Kafka consumers with different Group Id they will all read all partitions etc.

As to answer your question, C1 will read from offset 0 and C2 will read from offset 0 also, as they belongs to two different groups.

Answer 4

when consumers belong to different consumer groups they are supposed to consume the same messages as in a publish/subscribe pattern. when they are in the same consumer group, they are "competing" and reading on different partitions of the topic (assigned on subscription) so they don't get same messages. The consumer offset is saved in an internal __consumer_offset topic and the messages have the following format:

key = [group, topic, partition] value = offset

so as you can see, for different group(s) but same topic/partition, there is a different offset saved there.

Answer 5

Whenever a new ConsumerGroup is created and a consumer tries to read the data from a topic, the consumer always gets the data from the beginning of the topic (Offset 0).

Kafka maintains the offsets for each ConsumerGroup. Even if a consumer is in no ConsumerGroup, the consumer is added to a Kafka generated group in order to maintain the offsets of such a consumer. This process is carried out for all individual consumers.

In your case, consumer C2 will hence read the data from the offset 0.