Kafka (kafka-node) consumer group receives messages from all partitions

Question

I have one topic called "test-topic" with 3 partitions.

When I start a consumer (consumer-1) with group-id set to "test-group" it connects and reads from all partitions on the topic. So far so good.

The issue comes when I start another consumer (consumer-2) in the same group. I expect it to be re-balanced dividing partitions between the two consumers e.g. consumer-1 gets partition 0 and 2 and consumer-2 gets partition 1. This does not happen, sure I do get these re-balance callbacks but in the end, both consumers read from the same partition.

What is even more strange is that if I start another consumer that comes with Kafka (kafka-console-consumer.sh) then it will re-balance and divide all partitions between all consumers as expected.

I'm using kafka-node with the following options:

options = {
    groupId: 'test-group',
    kafkaHost: 'localhost:32769',
    protocol: ['range'],
    fromOffset: 'latest'
}

let topics = ['test-topic']

let consumer = new KafkaConsumerGroup(options, topics)

What I do see running the following command: kafka-consumer-groups.sh --bootstrap-server localhost:32769 --describe --group test-group --members --verbose

If I only run my two consumers from my nodejs project is one line with a consumer assigned to all partitions. But the CONSUMER-ID switches back and forth between my two clients.

When I start the command-line client (bin/kafka-console-consumer.sh --bootstrap-server localhost:32769 --topic test-topic --from-beginning --group test-group) it changes and all three consumers show up and are divided between each partition. As soon as I remove the kafka-console-consumer client it goes back to what I described earlier.

Have I misunderstood the concept of partitions and consumer groups or maybe I have the wrong configuration?

CrusaderX CrusaderX · Accepted Answer · 2019-12-13T19:07:33

I expect it to be re-balanced dividing partitions between the two consumers e.g. consumer-1 gets partition 0 and 2 and consumer-2 gets partition 1

You are right.

Have I misunderstood the concept of partitions and consumer groups or maybe I have the wrong configuration?

As I see you're doing all in right way. I think this is a bug in kafka-node library and I suggest you to try downgrade Kafka version (for instance, try this code with Kafka 2.1.0) if you're using the latest version (e.g 2.3).

Kafka (kafka-node) consumer group receives messages from all partitions

1 Answers