Why does the spring KafkaConsumer suspend all consumption from n topics when one fails to authorize

Question

I'm using the spring-kafka 2.3.6.RELEASE library to consume from a number of kafka topics. I'm using a single listener to consume all topics, as the interface allows. I've noted that, if a single topic fails to authorize, the KafkaConsumer instance suspends its thread, ceasing consumption from all topics until the retry interval elapses.

This seems very brittle to me, particularly for a technology focused on resiliency and parallelism.

Has anyone else observed this behaviour, and if so, is there any way to mitigate it beyond writing our own consumer implementation?

Answer 1

Spring simply delegates to the underlying kafka-clients jar so such things are out of its control.

You can use a separate listener for each topic. For example @KafkaListener is a repeatable annotation so you can add multiple annotations to a single method.