I am new to Flink. I have a Flink streaming program that counts something from kafka in 10 second session windows.
Here is my question:
Session windows default trigger is FIRE.
Will Flink streaming keep all kafka message of each window in memory?Or only keeps the result of aggregations.
If I use FIRE_AND_PURGE, what will be purged?
Firing and purging are described here, but simply put, the difference is that FIRE_AND_PURGE removes the window's contents after firing.
All of Flink's built-in window assigners (including session windows) take care of purging their contents when the time is right, i.e., after any allowed lateness has expired. But the purging is done as a separate step, rather than being combined with the trigger firing.
Windows' contents are kept in Flink state, so their location depends on which state backend you are using. This state will be kept on the heap if you are using the heap-based state backend, or on disk if you are using RocksDB.
Whether or not your windows are retaining their full contents until they are purged depends on whether you are using incremental aggregation -- in other words, if you are using reduce, aggregate, or fold, then only the ongoing result of the aggregation is being stored, otherwise it's a collection of all of the events assigned to the window.
