Multiple Consumers for Kafka topic

Question

I have implemented a kafka consumer similar to the way described in this article: http://howtoprogram.xyz/2016/05/29/create-multi-threaded-apache-kafka-consumer/

The way it is implemented implies only one consumer thread for one partition. So I want 10 consumer threads then I would need 10 topic partitions.

Are there any other working approaches for multithreaded consumers?

I alse looked into this article: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

However while testing this example in my environment I got an exception topic-1395414642817-47bb4df2 can't rebalance after 4 retries.

Answer 1

The way it is implemented implies only one consumer thread for one partition. So I want 10 consumer threads then I would need 10 topic partitions.

I assume you want to have more consumer thread than partitions? This is not supported by Kafka. In Kafka, each consumer monitors its progress (ie, what it did read the a partitions -- called its offset) itself. If you want more consumer threads than partitions, those threads would somehow need to talk to each other to "devide" the data. This pattern is non standard an not supported. It would also not scale very well, because if you add more consumers, the synchronization overhead would be high.

Having only one consumer thread per partitions keeps Kafka scalable. Best practice is to over partition your topics, to get more flexibility with regard to number of consumer threads.