I have one Kafka topic and five partitions for that one topic. There will be 5 consumer groups. Each consumer group has one service instances consuming from that topic.
Will the offset be the same in each consumer for the same record in Kafka?
4
votes
2 Answers
2
votes
By offset, if you mean the ordering of messages, then yes. It'd be the same for all consumers, because the ordering is determined by producers and brokers. So, if you have msg-1, msg-2, ..., msg-1000 in the topic, all the 5 consumers will consume those in that specific order. But the rate of consumption might vary. It has lots of variables (e.g. Network latency, network topology, consumer logic etc.) that determines the rate of consumption.
2
votes
The offset is assigned by the broker when the message comes into the partition so it's unique and it's not related to the consumers (and consumer groups). It identifies the unique position that the record has inside the partition. On the other side, each consumer (in a consumer group) reading from a specific partition will track its own offset which will be different from consumers (in other consumer groups); the offset concept in this case is used for tracking the position inside the partition from which reading messages. Of course it's always a message offset.
