Kafka multiple Consumers listening for multiple topics

Question

We are working on an application where we want to listen for more than one topics in Kafka. All the topics have a single partition. All the topic names will have a common prefix e.g. "test-x", "test-y" so we can use spring topicPattern for it.

We wanted to write a java spring consumer which listens to all the topics using regexp pattern. Our idea was, we can run multiple instances of the same consumer(belonging to the same group) and Kafka will distribute messages from different topics for different consumers.

However, This seems to be not working. Even if we are running multiple instances of the consumer only one consumer is getting messages from all the topics. If we create more than one partition in the topic then it works but that's something we won't have.

Is there anything which we have not understood here? Thanks in advance.

Consumers should be subscribed in different groups , in order for them to subscribe to same messages parallely — Kiran Balakrishnan
Sounds like strong Apache Kafka question in understanding the concept how topics are distributed when we use pattern subscription. Nothing to do from Spring level. I wonder if any Apache Kafka community member has ever hung on here in SO... — Artem Bilan
/me waves to @ArtemBilan ;) Plenty of us Kafka folk here :) Definitely sounds like a consumer group issue. — Robin Moffatt

Manjunath Manjunath · Accepted Answer · 2018-05-27T09:31:31

Kafka distributes partitions within a topic among a consumer group(consumers with same group.id). Since your topic has only partition, only one of the consumer in a consumer group would receive the message.

Kafka multiple Consumers listening for multiple topics

1 Answers