Kafka consumer in publish/subscribe mode

Question

I need to call Kafka consumer in publish/subscribe mode 1000 times. As far as I know for kafka to work in pub/subscribe mode I need to give a new groupId to each consumer( props.put("group.id", String.valueOf(Instant.now().toEpochMilli()));). But when I do this if two consumer threads access consumer at the same millisecond there will be problems. How should this problem be solved?

The only requirement is to have group.id unique. You can use UUID.randomUUID().toString() — Katya Gorshkova
A consumer is only a subscriber, not a publisher... Unclear what exactly you're looking for. Are you making 1000 consumers of one group? Does your topic have 1000 partitions to support that? — OneCricketeer
@cricket_007 To use kafka in a queue mode you need to define all consumers in the same group. To use Kafka in pub/sub mode you need to define each consumer in a different group as far as I know. Take a look at the documentation in the below answer. — neb

Giorgos Myrianthous Giorgos Myrianthous · Accepted Answer · 2018-05-13T10:10:27

If you want to spread the messages across the consumers you need to use the same group.id. If you have 1000 messages and 1000 consumers, then each of the consumer will normally consume one message.

On the other hand, if you want each of the consumer to consume all the messages from the topics, you need to use a different group.id so that the messages in the topic are consumed by all consumers. If you have a huge number of consumers you can use UUID.randomUUID().toString() in order to produce a distinct group.id for each one.

According to the docs:

Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.

If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.

If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.

Kafka consumer in publish/subscribe mode

1 Answers