I am using aggregateFunction to count on unique objects in a time window of 10mins. I wanted to enable checkpoints for the window function, just in case if the job goes down, we don't lose any accumulated states. I've been reading on it, include this post (very helpful). But I still don't quite understand how should I utilized the ProcessWindowFunction to keep some state variable, so that when my job goes down and restart, I don't lose any accumulated data?
0
votes
1 Answers
0
votes
In the case you describe, you don't have to do anything. All of the state that needs to be checkpointed is already being managed by Flink.
The global state described in that other post is something you would use if you needed to store results so they could be referred to when processing later windows. But in your case, it sounds the results for one 10 minute window don't depend on the results for earlier windows.
If you want to get a hands-on feel for how Flink's fault tolerance really works, there's a docker-based playground that makes it easy to do some experiments and directly observe what happens: see the Flink Operations Playground. The application involved does windowing, so you should find it nicely relevant.
