Should the complete latency of a Storm Topology be equal (or close) to the sum of the bolts?

0
votes

We have a storm topology that usually runs fine. We have no errors in the logs and times are fast. However, sometimes we get spikes in the "complete latency"

enter image description here

Here are the individual bolts:

enter image description here

So my question is: What causes the Complete latency to be much more than the individual bolts? ( We are using the Microsoft spout and event hubs if that helps).

1
azureapache-storm

1 Answers

1
votes

No it's not. Latencies for each component excludes queue wait time and transfer latency between worker. 'complete latency' means all the nodes in tuple tree are acked, so it reflects slowest path of the tree.

Btw, behind the scene, 'complete latency' includes waiting time for spout to handle ack from acker, and if your spout spends long time in nextTuple it will heavily affect 'complete latency'. It is fixed from STORM-1742 and will be included to next release.

Hope this helps.