Sarama Partition Consumer with ConsumerGroup

Question

I have tried shopify/sarama library to consume kafka messages. I used both Consumer interface and ConsumerGroup interface. I can consume from specific partitions using ConsumePartition() method in Consumer. But when I use ConsumerGroup interface, I do not seem to have the capability to consume from a specific partition.

Is there a way for me to assign certain partitions to specific consumers inside a consumer group? Or is it something I cannot interfere with?

Try using the assign() method of the KafkaConsumer. You can assign a specific partition using this. — Fatema Khuzaima Sagar
@fatemasagar assign() method is in confluent kafka library, right? There is no such method in sarama library. — Dean Karstle
Yes, Dean, you are correct about that. Sorry for that answer. — Fatema Khuzaima Sagar
The Java API allows assigning within a consumer group. I'd be surprised if sarama didn't — OneCricketeer
this strange to me as well...not aware much with sarama however this may be benfit to you stackoverflow.com/questions/44279935/… — Nitin

Dean Karstle Dean Karstle · Accepted Answer · 2020-01-05T13:50:12

It seems like I cannot give exact partitions to consume when using ConsumerGroup. However I can choose the strategy that I want to assign partitions to my consumers, out of 2 strategy options.

Balance Strategy Range

BalanceStrategyRange is the default and assigns partitions as ranges to consumer group members. Example with one topic T with six partitions (0..5) and two members (M1, M2):

M1: {T: [0, 1, 2]}
M2: {T: [3, 4, 5]}

Balance Strategy Round-Robin

BalanceStrategyRoundRobin assigns partitions to members in alternating order. Example with topic T with six partitions (0..5) and two members (M1, M2):

M1: {T: [0, 2, 4]}
M2: {T: [1, 3, 5]}

I can give this as a configuration when creating the ConsumerGroup.

config.Consumer.Group.Rebalance.Strategy = BalanceStrategyRange

Sarama Partition Consumer with ConsumerGroup

1 Answers