Are Kafka partitions consumed evenly?

Question

I have a consumer group with several consumers. Each consumer is assigned to a set of partitions. When the consumer polls for messages where the consumed partition is selected? Is it done on the consumer side or does Kafka server decide which partitions turn it is to get consumed?

Some of my partitions have a lot of messages, but some have none or very little. But I still need my consumers to consume each of it's assigned partitions equally. So I need my consumer to loop through the partitions fast, preferably poll x messages from each assigned partition.

I'm using https://github.com/appsignal/rdkafka-ruby in case it matters.

Answer 1

Kafka assigns the partitions to be consumed as a Round-Robin strategy giving to each partition a fair chance for consumption. In that way starving for the partitions is avoid.

On the other hand, Kafka It does not guarantee that the data will be consumed proportionally across the partitions,

Please see the details about this here.