In Storm Spout, Naming the Consumer Group

Question

I am currently using: https://github.com/wurstmeister/storm-kafka-0.8-plus/commits/master which has been moved to: https://github.com/apache/storm/tree/master/external/storm-kafka

I want to specify the Kafka Consumer Group Name. By looking at the storm-kafka code, I followed the setting, id, to find that is is never used when dealing with a consumer configuration, but is used in creating the zookeeper path at which offset information is stored. Here in this link is an example of why I would want to do this: https://labs.spotify.com/2015/01/05/how-spotify-scales-apache-storm/

Am I correct in saying that the Consumer Group Name cannot be set using the https://github.com/apache/storm/tree/master/external/storm-kafka code?

Adrian Seungjin Lee Adrian Seungjin Lee · Accepted Answer · 2015-04-23T05:30:48

So far, storm-kafka integration is implemented using SimpleConsumer API of kafka and the format it stores consumer offset in zookeeper is implemented in their own way(JSON format).

If you write spout config like below,

SpoutConfig spoutConfig = new SpoutConfig(zkBrokerHosts,
                "topic name",
                "/kafka/consumers(just an example, path to store consumer offset)",
                "yourTopic");

It will write consumer offset in subdirectories of /kafka/consumers/yourTopic.

Note that by default storm-kafka uses same zookeeper that your Storm uses.

In Storm Spout, Naming the Consumer Group

1 Answers