Flink timer async

Question

I have a question regarding Flink and its timer service.

I have a keyBy stream which uses a timer, When the timer is called I need to send an http request which might take time to respond. My question is, should I make the http call async? or flink is making the timer call already as a new thread with async per key?

Thanks in advance

Answer 1

You can use a ProcessFunction that stores the data required for the HTTP request, and that can have a timer. When it fires, you emit a record that has the request data, which a subsequent AsyncFunction will use to make the periodic request that you need.

Answer 2

If You are asking if the onTimer method is invoked in a separate thread for each key, then I am pretty sure that it is not. So, You would need to invoke the HTTP call asynchronously in this case.

But to be completely honest, I don't think this is a good idea, in general, to use the onTimer function to perform HTTP calls. I don't know anything about Your use-case, but I think You should consider using different mechanisms like creating side-output and then using the Flink Async operator. Using asynchronous calls inside the onTimer can be tricky, especially if You consider things like retries, timeouts and possible failures.

So according to comment the use-case is to make a call to service each X mins and then post something to Kafka. So, what You could do is to simply have a process function that schedules timers. Each time the timer is fired You then generate some output record with data needed for request if there is any data needed. Then You use the Async operator to actually perform the requests, parse the response and based on the response generate some output record that can be then saved to Kafka.