In our storm 1.0.2 application we are facing out of memory exceptions.Upon debugging we saw that the Kafka spout was emitting too many messages to the bolts . The bolts were running at a capacity of almost 4.0. So is there a way to enable backpressure in storm so that the spout emits depending on the capacity in bolts. Tried enabling the topology.backpressure.enable to true but ran to this issue https://issues.apache.org/jira/browse/STORM-1949. We are using the out of the box implementation of KafkaSpout and extending the BaseRichBolt for our bolts .Our DAG is linear.
3
votes
1 Answers
7
votes
You can handle the back pressure of the KafkaSpout by setting the maxSpoutPending value in the topology configuration,
Config config = new Config();
config.setMaxSpoutPending(200);
config.setMessageTimeoutSecs(100);
StormSubmitter.submitTopology("testtopology", config, builder.createTopology());
maxSpoutPending is the number of tuples that can be pending acknowledgement in your topology at a given time. Setting this property, will intimate the KafkaSpout not to consume any more data from Kafka unless the unacknowledged tuple count is less than maxSpoutPending value.
