How interpret Storm UI statistic

Question

I have some problems with interpreting storm UI stats. I've deployed a simple topology with Kafka spout and Custom Bolt that do nothing, but acknowledge tuples. The idea was to measure performance of Kafka spout.

So, looking at this stats I've got some questions.

1) In last 10 min topology acked 1619220 tuples and complete latency was 14.125ms. But if you make some calculations:

    (1619220 * 14.125) / (1000 * 60) = 381

1619220 tuples with complete latency 14.125 each, requires 381 min to pass topology. But that stats was on last 10 minutes. Complete latency shows wrong number, or I interpret it wrong?

2) Bolt capacity is around 0.5. Does it proof that bottleneck is kafka spout?

I will appreciate any information about improving storm topology perfomace, since it's not obvious for me right now.

Answer 1

As tuples processing does overlap for many tuples in-flight due to parallelism, your calculation is not correct. (It would only be correct, if Storm would process a single tuple completely before starting with the next tuple.)

A capacity on 0.5 is not really an overload. Overload would be more than 1. But I would suggest to try to keep it below 0.9 or maybe 0.8 to get some head room for spikes in you input rate.