I've been running a performance test on our Apache Storm topology and noticed that the 'Failed' count from the kinesis-spout to be very high (almost 1/3 of the Tuples acked). What is this value?
Are they the Tuples that failed to be processed end to end in the topology within the timeout value. Is there a way to verify that the Failed count was due to a processing timeout?
Yes, the failed count shows the number of failed (ie, not end-to-end processed) tuples. That a tuples fails can have two causes. Either a tuples is failed by the user code (by calling Collector.fail(...)) or if a tuple times out.
Thus, if there is no manual failing involved, the cause for failing tuples is a time out. The default time out in Storm is 30 seconds and can be configured via Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS.
