how to understand consumer and listener in spring kafka

Question

I using spring kafka with multi-thread feature(ConcurrentKafkaListenerContainerFactory), I found 2 types of thread names like this: 1. #0-1-kafka-consumer-1 2. #0-1-kafka-listener-3

so how can I understand these 2 kinds of thread? what's the relationship among them? Thanks in advance!

Answer 1

The consumer thread polls the KafkaConsumer for messages and hands them over to the listener thread which invokes your listener.

This was required with early versions of the KafkaConsumer because a slow listener could cause partition rebalancing - the heartbeats had to be sent on the consumer thread.

They have now fixed this in the KafkaConsumer (heartbeats are sent in the background) so in 2.0 we will only have one thread type and the listener is invoked on the consumer thread. 2.0.0.M2 (milestone 2) is available now; the release is planned for around the end of next month.

Answer 2

In the previous Kafka version (< 0.10.1) that was a pain to have slow listeners. Non-active consumer thread considered as dead and therefore rebalance happened. That's why we introduced thread hands off and delivered records for processing in the listener to the separate thread. So, those thread prefixes are exactly about that.

In the latest version 2.0, based on the Kafka 0.10.2 we have removed that logic because now heartbeat happens in the Kafka client itself and properly. Therefore we don't need to worry about slow listeners any more - everything now works on the consumer's thread.