Why partition is needed in Shopify Sarama consumer to consume messages

2
votes

I am sorry for posting a question related to a Kafka Library as not many people are interested in Library specific questions. But this library is one of the most used library for golang-Kafka implementations.

I want to create a simple consumer using Sarama library which listens to a topic. Now as far as I know, in the high-level Kafka API's, by default a consumer listens to all the topics partitions if a specific partition is not specified. However, in this Library, the Consumer interface has only ConsumePartition function where the partition is required param. The signature of function is:

ConsumePartition(topic string, partition int32, offset int64) (PartitionConsumer, error)

This confuses me a bit. Anyone who has worked on it?

Also, I have a basic question regarding Kafka. If I have a consumer group consisting of 3 consumer instances and they are listening to let's say 2 topics each having 2 partitions, then do I need to specifically mention which consumer instance will consume to which partition or Kafka Fetch API will take care of it on its own based on load?

2
goapache-kafkasarama
no update on this? @Rishabh - Ivan Aracki

2 Answers

3
votes

I use sarama-cluster which is an open source extension for Sarama (also recommended by Shopify Sarama).
With Sarama cluster you can create a consumer using this API:

cluster.NewConsumer(brokers, consumerGroup, topics, kafkaConfig)

so no partition is needed. You should only provide the addresses of your Kafka brokers, the name of your consumer group and which topics you wish to consume.

Consumers handling

To maintain order you should assign to each partition only one consumer.
So in case you have 3 consumers in your consumer group and you want them to consume 2 topics having 2 partitions each, you should assign as follows: 

partitions 1,2 -> consumer A  
partition 3 -> consumer B  
partition 4 -> consumer C

You might end up with one of the consumers advancing faster (one of the topics have higher throughput) and you will need to re-balance.
Using a library (like sarama-cluster) that handles this for you is recommended.

0
votes

Meanwhile the Sarama cluster project has been deprecated, see the Deprecation Notice in their repo. The good news it has been deprecated in favor of the PR Implement a higher-level consumer group #1099, which focusses on consuming topic(s) rather than dedicated partitions.

The examples/ folder in the official sarama repo provides a good Consumer Group Implementation Example. It works like a charm and doesn't require any additional libraries on top of Sarama.

Preview:

config := sarama.NewConfig()
client, err := sarama.NewConsumerGroup(strings.Split(brokers, ","), "mygroup", config)
client.Consume(ctx, strings.Split(topics, ","), &consumer) // should run in a loop