kafka consumer is not picking up the message from topic even though previous message manually committed

Question

I have two messages in Kafka Topic (let's say offset 1 and 2) and both messages are placed in same partition (let's say p1).

My consumer app is like this:

my consumer is picking up the message 1 (with offset 1 from patition 1) and sending manual commit signal to Kafka and then waiting for 5 secs.

My expectation is since commit signal went to kafka, while my thread 1 is waiting for 5 secs and another consumer thread should pickup the message 2 from partition 1 and process it in separate thread.

However, it is not working like this. it is processing one after the other. only after 5secs finished by thread 1 then it is picking up the second message from topic.

NOTE: I have made sure that amount of inparallel consumers are set to more than one (in my case 5 and max consumer pool size is 10).

Am I doing anything incorrect? has anyone faced similar issue? if so, what is the solution?

thanks, Bala

Answer 1

Each Partition can be consumed only by one thread at a time and that thread will continue to wait(there are other factors), until unless a rebalance is triggered, which then assigns that partition to a different thread.

Rebalance will be triggered

• Either manually or
• When new thread is added with same consumer group or
• When one of the threads stop calling poll method for max.poll.interval.ms many milliseconds (by default is 5 mins)

Here is blog with lot more details about it.