Storm Spout / Topology Performance

Question

I am having an issue with Apache Storm performance, mainly from the spout.

I have a topology that emits items from a kestrel queue. I fetch about 2000 items and each time nextTuple is called in the spout I emit one.

When running in a local cluster on my i7 Macbook Pro I see that I emit around 20 tuples per second, and that storm will call nextTuple once every 50ms.

I am running using 1 spout task, and 1 spout executor. I have setMaxSpoutPending set to 10.

Why is there such a large time gap between each call to nextTuple? Is the outputCollector waiting to hear back from each tuple before emiting a new one?

I am running java 8 and storm version 0.9.4

Answer 1

from documentation: https://storm.apache.org/apidocs/backtype/storm/spout/ISpout.html

Storm executes ack, fail, and nextTuple all on the same thread. This means that an implementor of an ISpout does not need to worry about concurrency issues between those methods. However, it also means that an implementor must ensure that nextTuple is non-blocking: otherwise the method could block acks and fails that are pending to be processed.

Tip 1: emit 1 tuple when nextTuple is called. Tip 2: don't fetch data in nextTuple, do it in a separate thread and use any concurrent queue to push and poll the data so nextTuple will only poll. Tip 3: try setting setMaxSpoutPending to 1. tip 4: make sure each bolt's execute(tuple) method is optimized.

Answer 2

Your pattern is quite off... You should either fetch only a single tuple from the queue, or emit all 2000 tuples you fetched. This would follow the design of Storm.

Using an extra thread does not align with Storm's design. And there is no reason why you would not want to emit all 2000 tuples at once...

And yes, if you assign message IDs in your Spout and set max.spout.pending Storm will not do the next call to Spout.nextTuple() as long as more than 10 tuples are in-flight, ie, not acked.