Apache Storm: Support of topic wildcards in Kafka spout

Question

We have a topology that has multiple kafka spout tasks. Each spout task is supposed to read a subset of messages from a set of Kafka topics. Topics have to be subscribed using a wild card such as AAA.BBB.*. The expected behaviour would be that all spout tasks collectively will consume all messages in all of the topics that match the wild card. Each message is only routed to a single spout task (Ignore failure scenarios). Is this currently supported?

Answer 1

Perhaps you could use DynamicBrokersReader class.

Map conf = new HashMap();
...
conf.put("kafka.topic.wildcard.match", true);

wildCardBrokerReader = new DynamicBrokersReader(conf, connectionString, masterPath, "AAA.BBB.*");
List<GlobalPartitionInformation> partitions = wildCardBrokerReader.getBrokerInfo();
...
for (GlobalPartitionInformation eachTopic: partitions) {
    StaticHosts hosts = new StaticHosts(eachTopic);
    SpoutConfig spoutConfig = new SpoutConfig(hosts, eachTopic.topic, zkRoot, id);
    KafkaSpout spout = new KafkaSpout(spoutConfig);
}
... // Wrap those created spout instances into a container