messages published to all consumers with same consumer-group in spring-data-stream project

Question

I got my zookeeper and 3 kafka broker running locally. I started one producer and one consumer. I can see consumer is consuming message.

I then started three consumers with same consumer group name (different ports since its a spring boot project). but what I found is that all the consumers are now consuming (receiving) messages. But I expect the message to be load-balanced in that only messages are not repeated across the consumers. I don't know what the problem is.

Here is my property file

spring.cloud.stream.bindings.input.destination=timerTopicLocal
spring.cloud.stream.kafka.binder.zkNodes=localhost
spring.cloud.stream.kafka.binder.brokers=localhost
spring.cloud.stream.bindings.input.group=timerGroup

Here the group is timerGroup.

consumer code : https://github.com/codecentric/edmp-sample-stream-sink

producer code : https://github.com/codecentric/edmp-sample-stream-source

Marius Bogoevici Marius Bogoevici · Accepted Answer · 2016-09-29T04:59:00

Can you please update dependencies to Camden.RELEASE (and start using Kafka 0.9+) ? In Brixton.RELEASE, Kafka consumers were 0.8-based and required passing instanceIndex/instanceCount as properties in order to distribute partitions correctly.

In Camden.RELEASE we are using the Kafka 0.9+ consumer client, which does load-balancing in the way you are expecting (we also support static partition allocation via instanceIndex/instanceCount, but I suspect this is not what you want). I can enter into more details on how to configure this with Brixton, but I guess an upgrade should be a much easier path.

messages published to all consumers with same consumer-group in spring-data-stream project

1 Answers