How to execute functions on empty windows in Flink streaming?

Question

I wrote a Flink program that calculates the number of events per keyed window from a simple kafka stream. I works great, fast & accurate. When the source stops, I would like to have 0 as result of the calculation on each window, but no result is sent. The function just does not execute. I assume this is because of the lazy operation behavior of Flink.

Any recommendation?

I'm not sure, whether I understand you correctly. Do you want that windows are triggered even though they don't contain an element? If this is the case, then this is not possible at the moment. The problem is that you would have to trigger a window for every possible key value. — Till Rohrmann
Do they emit for each key an empty window? Let's say we have a long as key, then we'll have to deal with 2^64 windows (as an upper bound)? — Till Rohrmann
Thanks! Right now we,re solving it on destiny, inserting the missing timestamp elemebñnts of the serie with Elasticsearch (we're generating metrics from some feeds). It works, but could be great if flink had something like 'forced windows' in a global config way. — Sergio Perea

Hadrien Lepousé Hadrien Lepousé · Accepted Answer · 2018-12-01T11:45:20

I encountered the same situation. Filling the holes in your database with another process is a solution.

However, I found it easier to union your main stream with a custom periodical source, that emits dummies, whose only roles are to trigger windows creation. When doing this, you have to make sure that dummies are ignored in computations.

Here is how to code a periodical source (however you may not need a RichParallelSourceFunction, a SourceFunction can be enough)

How to execute functions on empty windows in Flink streaming?

1 Answers