Decreasing max.spout.pending value leads to failed messages in Kafka Spout in Storm UI?

Question

We are trying to benchmark the performance in our Storm Topology. We are ingesting messages around 1000/seconds to Kafka Topic. When we put max.spout.pendind=2000 in our KafkaSpout then we don't see any failed messages in storm UI but when we decrease the max.spout.pendind value to 500 or 100, then we see lot of failed messages in spout in Storm UI. My understanding is that if we keep the max.spout.pending low then we will not have any failed messages as nothing will timeout but it behaving in opposite manner. We are using Storm 1.1.0 version from HDP 2.6.5 version.

We have one Kafka Spout and two bolts .

KafkaSpout Parallelism               - 1
Processing Bolt Parallelism          - 1
Custom Kafka Writer Bolt Parallelism - 1

Could anyone have any idea about this?

Answer 1

The first you will have to do is on the storm UI check the statistics of the latency. You should also look at how the bolts/spouts are loaded (capacity statistics). Is the rate of emit of tuples really high compared to the rate of sinking this data ? , This is an indication that i get when you mention that increasing pending spouts is fixing the issue. Can you provide these stats .. Another part worth exploring is increasing the task time out on the tuples (to see if this is causing replay and flooding the topology)

Answer 2

Please find the below topology stats :

Answer 3

This is interesting. You are right, follow my steps to narrow down the issue,

1. Upload a screenshot of your Topology Visualization screen on peek load.
2. Check for the bolts which are changing their color to brown/red. Red is indicating that your bolt takes too much time to process records.
3. Your spout/bolt executors are way less to process 1K tuples per second.
4. Number of machines you are using?
5. If tuples are failed in "KafkaSpout" then most of the time it means timeout error.
6. Find out after processing how many events tuples are failing.