0
votes

This is a two question topic about flink streaming based on experiments I did myself and I need some clarification. The questions are:

  1. When we use windows on a KeyedStream in flink, are the computations of the apply function asynchronous? Specifically, will flink create separate windows per key and process these windows independently from one another?

  2. Assume that we use the apply function (do some computations) on a windowed stream which will then create a DataStream. If we do some transformations on the resulting DataStream, will flink hold the entire WindowedStream in memory? And will flink wait until all the apply functions of the WindowedStream are finished and then move on to the transformations on the resulting stream?

In all the experiments I did I used event time and I read the data from a file. I have observed the above statements in my experiments and I need some clarification.

1

1 Answers

1
votes

Ad. 1 Yes, each key is processed independently. It is also the way windows computations are parallelised.

Ad.2 Flink will keep windows state until the window can be emitted (plus some extra time in case of allowedLateness). Once results for a window are emitted(in your case are forwarded to next operator), the state can be cleared.