Storm bolt following a kafka bolt

Question

I have a Storm topology where I have to send output to kafka as well as update a value in redis. For this I have a Kafkabolt as well as a RedisBolt. Below is what my topology looks like -

tp.setSpout("kafkaSpout", kafkaSpout, 3);

tp.setBolt("EvaluatorBolt", evaluatorBolt, 6).shuffleGrouping("kafkaStream");

tp.setBolt("ResultToRedisBolt",ResultsToRedisBolt,3).shuffleGrouping("EvaluatorBolt","ResultStream");

tp.setBolt("ResultToKafkaBolt", ResultsToKafkaBolt, 3).shuffleGrouping("EvaluatorBolt","ResultStream");

The problem is that both of the end bolts (Redis and Kafka) are listening to the same stream from the preceding bolt (ResultStream), hence both can fail independently. What I really need is that if the result is successfully published in Kafka, then only I update the value in Redis. Is there a way to have an output stream from a kafkaBolt where I can get the messages published successfully to Kafka? I can then probably listen to that stream in my RedisBolt and act accordingly.

Why can't you have your RedisBolt listen to KafkaBolt? KafaSpout->EvaluatorBolt->KafkaBolt->RedisBolt — Ajeesh
I was looking for out of box support for KafkaBolt to emit values which is not there. As per suggested answer, there is a possibility to add producer callback to emit now. — Harsh Kumar

Stig Rohde Døssing Stig Rohde Døssing · Accepted Answer · 2018-09-09T08:31:12

It is not currently possible, unless you modify the bolt code. You would likely be better off changing your design slightly, since doing extra processing after the tuple is written to Kafka has some drawbacks. If you write the tuple to Kafka and you fail to write to Redis, you will get duplicates in Kafka, since the processing will start over at the spout.

It might be better, depending on your use case, to write the result to Kafka, and then have another topology read the result from Kafka and write to Redis.

If you still need to be able to emit new tuples from the bolt, it should be pretty easy to implement. The bolt recently got the ability to add a custom Producer callback, so we could extend that mechanism.

See the discussion at https://github.com/apache/storm/pull/2790#issuecomment-411709331 for context.

Storm bolt following a kafka bolt

1 Answers