I am building a Spring Application in Java 9 with Apache Storm 1.1.2 and Kafka 0.11
I've noticed that under a high load (2500 messages per second) that the kafka spout has a very high lag. The Kafka Spout has a parallelism hint of 3. The lag is nearly equal to the Spout Committed Offset.
This lag sets a cap on the maximum amount of messages the topology can ingest per second which is not great. Does anyone know a solution for this?
UPDATE: I've also noticed that even with 10 workers and a parallelism hint of 4, the load is not divided between different kafka spout executors from what I can tell in the UI. The topic has 8 partitions
