Apache Flink timeout using onTimer and processElement

Question

I am using the Apache Flink processElement1, processElement2 and onTimer streaming design pattern for implementing a timeout use case. I observed that the throughput of the system decreased by orders of magnitude when I included the timeout functionality.

Any hint on the internal implementation of the onTimer in Flink: is it one thread per key stream (unlikely), or a pool/single execution threads that continuously polls buffered callbacks and pick up the timed-out callbacks for execution.

To the best of my knowledge, Flink is based on the actor model and reactive patterns (AKKA) which encourages the judicious usage of few non-blocking threads, and hence one thread per key stream for onTimer or any other pattern is typically not used!

Answer 1

There are two kinds of timers in Flink, event time and processing time timers. The implementations are completely different, but in neither case should you see a significant performance impact. Something else must be going on. Can you share small, reproducible example, or at least show us more of what’s going on and how you are taking the measurements?