Turn off acking in KafkaSpout while respecting maxSpoutPending

Question

I'm using storm 0.9.3. I'm trying to turn off acking per tuple for my topology. I set Config.TOPOLOGY_ACKER_EXECUTORS to 0, and maxSpoutPending to 500. When I run my topology, I'm noticing that maxSpoutPending is being ignored and the spout continues emitting well past that limit. Here's my config -

config.setNumWorkers(3);
config.setMaxSpoutPending(500);
config.put("topology.sleep.spout.wait.strategy.time.ms", 50);
config.put("topology.message.timeout.secs", 300);
config.put(Config.TOPOLOGY_ACKER_EXECUTORS, 0);

I'm using KafkaSpout to read from Kafka and a single bolt to consume the message.

"When I run my topology, I'm noticing that maxSpoutPending is being ignored and the spout continues emitting well past that limit." -- Yes, this is actually the expected behavior (though this may not be what you want I understand). Once Storm will support back pressure the situation will improve for use cases such as yours. — Michael G. Noll

Shams Shams · Accepted Answer · 2015-02-21T21:24:45

By setting TOPOLOGY_ACKER_EXECUTORS to 0, storm will acknowledge all tuples immediately when they come off the spout, which may not be reliable, because no mechanism will work to check if tuple is processed or failed.
And by setting setMaxSpoutPending tells storm the maximum number of tuples pending on spout to process. MaxSpoutPending dosen't limit your output. If you want to see real output frequency of your storm topology, check for topology latency for your running storm topology in Storm UI.

Turn off acking in KafkaSpout while respecting maxSpoutPending

1 Answers