Create multiple consumers for same topic in kafka

Question

I am newbie and can see one example with one consumer in below github repository, but any ideas how to create multiple consumers for same topic in go lang?

https://github.com/confluentinc/confluent-kafka-go/tree/master/examples

Any consumer factory (to generate N consumers) available in confluent-kafka to read same topic (with partitions)?

@Ullaakut I am checking if any consumer factory would be available to create N number of consumers for same topic (with different partitions) in confluent kafka? Updated my question. — user923499
Again, what have you tried, though? For example, try to create a list of consumer objects and assign them all a given partition? Generally, one application only runs one consumer, though. Then you scale your application independently — OneCricketeer
@cricket_007, any reason why one application should run one consumer. I see having partitions can improve the data processing time. I was able to get the partition count in topic, create multiple consumers and read the data from different partitions and it worked with small data. Just need to tweak code in this example, github.com/confluentinc/confluent-kafka-go/blob/master/examples/… — user923499
For simplicity? Scalable microservices? Rather than having one machine become a bottleneck, you distribute it over many. Otherwise, one application is already assigned all the partitions for a given topic. If you actually need two distinct topics, I've only seen that being done in Kafka Streams were you would first join the topics on some field — OneCricketeer

Nishu Tayal Nishu Tayal · Accepted Answer · 2018-09-24T19:57:00

There is an example in the Confluent github repo :

https://github.com/confluentinc/confluent-kafka-go/blob/master/examples/consumer_example/consumer_example.go

If you want to create multiple consumers for the same topic, there are two scenarios :

1.Create each consumer with different group id.

c1, err := kafka.NewConsumer(&kafka.ConfigMap{
        "bootstrap.servers":    broker,
        "group.id":             group1,
        "session.timeout.ms":   6000,
        "default.topic.config": kafka.ConfigMap{"auto.offset.reset": "earliest"}})

c2, err := kafka.NewConsumer(&kafka.ConfigMap{
        "bootstrap.servers":    broker,
        "group.id":             group2,
        "session.timeout.ms":   6000,
        "default.topic.config": kafka.ConfigMap{"auto.offset.reset": "earliest"}})

If you want multiple consumers for the same topic but with the same group id, it will start consuming from single partition. i.e. A topic contains 3 partitions and you create 3 consumers with same group id, each consumer will consume from one partition

Create multiple consumers for same topic in kafka

1 Answers