Consuming multiple kafka topics in the same consumer class

Question

I have a spring boot project and I am spring-kafka to connect to the underlying kafka event hub.

I have to listen to 2 different topics in the same consumer class. I have two approaches to do so.

One is to have two kafka listeners like this:

 @KafkaListener(topics = "topic1")
public void consumeTopic1(String message) throws Exception {  
   //do something
  }

 @KafkaListener(topics = "topic2")
public void consumeTopic2(String message) throws Exception {  
   //do something
  }

Another approach is to have 2 topics in the same kafkaListener like this

 @KafkaListener(topics = {"topic1", "topic2"})
public void consumeTopics(String message) throws Exception {  
   //do something
  }

Since I am new to kafka, I am not sure what is the difference between the two approaches. Which one is performant and resource effective.

One thing I am wondering about is that would it listen to both the topics on a single thread in both approaches or it would spawn a thread each to listen to these topics.

Using approach 1, I have had troubles with the consumer where I see some delays in the consumption of the topics.

Please suggest me based on your experience as I am pretty new to kafka

===============EDIT======== Kafka properties are in application.yml as follows:

 kafka:
    properties:
      topics:
        topic1: topic1
        topic2: topic2
    bootstrap-servers: server1,server2
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      retries: 4
    consumer:
      group-id: mygroupid
      auto-offset-reset: latest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

I actually don't have a consumerConfig. I just have a class marked with a service annotation in which I have two methods with KafkaListener annotations. Do I need to load some special consumer beans? — Hary
Possible duplicate of what is best practice to consume messages from multiple kafka topics? — mukesh210
at least can you show the properties in application.yml or application.properties — Deadpool
Sure. I just edited my post with application.yml portion pertinent to kafka. Please take a look — Hary

Anirudh Simha Anirudh Simha · Accepted Answer · 2019-05-08T12:49:46

The purpose of creating multiple topics is to spread the data, and
subsequently, the processing of data, across multiple
cores/processes/threads.
Even then, you can start with a single consumer for multiple topics until a delay is observed in processing events in queue. Then you can separate them out.

Consuming multiple kafka topics in the same consumer class

1 Answers