Creating multiple consumers for a Single kafka topic

Question

I have a spring boot application. I wish to implement a functionaly that consumes from a Kafka topic and for every consumed message it calls another method that does some processing on the message and then stores it in a database. The source kafka topic has 12 partitions. I want to create 6 or 12 kafka consumers under the same consumer group. All of the consumers will consume messages from Kafka and call the function. I am not able to get to a good design for this. Can someone help me ?

As usual, Kafka consumers will consume the same messages from the topic and process them for a different purpose. If you just process a message and store it into the same database, I think you just need to create a single consumer. Am I understanding your post correctly? — Tran Ho
So If I am creating all the consumers with the same consumer group. The consumers will be divided among the partitions. So if I am creating 12 consumers every consumer will consume from only 1 partition. So with more number of consumers I will be able to consume from partitions in parallel and processing will be faster. — Anuja Barve
it calls another method that does some processing on the message and then stores it in a database. - please use Kafka Connect for this. Don't use Spring for writing to a database — OneCricketeer

EFOE EFOE · Accepted Answer · 2020-02-19T13:00:30

We have pretty the same need for a topic with 4 partitions, we created a consumer group with 2 members, and we assigned each member ( instance of our consumer) two partitions. If you are using spring you can select how many members will be created by instances with this parameter:

# For Kafka concurrency 
spring.kafka.listener.concurrency=2

If you are using the Kafka library this parameters should be in the parameters of your consumer. Be aware to not exceed you number of threads availables.

Creating multiple consumers for a Single kafka topic

2 Answers