Only one consumer gets kafka messages with 5 partitions

0
votes

We tested consumers group with 2 and 3 consumers (knowing that we may have more consumers in the future) using the confluent kafka package for Go.

With 10 partitions per topic the messages are split between all the consumers.

With 5 partitions per topic somehow only one consumer gets the messages.

Any idea why this behavior?

1
goapache-kafkaconfluent-platform
Are the messages spread evenly across partitions? - Thilo
How can I check this? From my logs it looks like it - stop-start
How do you set messages? Do your messages have key set, or it's only value& - streetturtle
@streetturtle: No we don't use key set. - stop-start
You can debug the assignment process by setting the debug property to cgrp - Edenhill

1 Answers

0
votes

Can you try using this command to see the partition assignments, and add the results here? bin/kafka-consumer-groups.sh --bootstrap-server <your-broker-ip>:9092 --describe --group <your-consumer-group> --members --verbose

By default, Kafka uses range partitioning scheme, so sometimes there are unexpected behaviors (e.g. if you have only one partition per topic, and multiple consumers, only one consumer will be assigned to all the partitions). I don't know if that's relevant here, but you can take a look at this post for more detail.