One partition multiple consumers same group, consumer IDs

Question

We have one topic with one partition due to ordering of message requirements. We have two consumers running on different servers with same set of configurations i.e. groupId, consumerId, consumerGroup. i.e.

1 Topic -> 1 Partition -> 2 Consumers

When we deploy consumers same code is deployed on both the servers. Noticed when a message comes we see both the consumers are consuming message rather than only one processing. Reason having consumers running on two separate servers is if one server crashes at least other can continue processing messages. But looks like if both up both consuming messages. Reading Kafka docs it says if we have more consumers than partitions then some stay idle don't see that happening. Anything we are missing on configuration side apart from consumerId & groupId. Thanks

Any chances to see some configuration on the matter and some logs from both consumer instances confirming that we have a problem you describe. As far as we use the same consumer group, only one instance should consume - all the rest stay idle. — Artem Bilan
They either have different group.ids somehow, or you must be assigning the partition manually rather than letting Kafka distribute it using group management. — Gary Russell

Martin Dengler Martin Dengler · Accepted Answer · 2021-02-11T19:56:43

As @Gary Russel said, as long as the two consumer instances have their own consumer group, they will consume every event that is written to the topic. Just put them into the same consumer-group. You can provide a consumer-group-id in the consumer.properties.

One partition multiple consumers same group, consumer IDs

1 Answers