Problem Definition & Establishing Concepts
Let’s say we have a TumblingEventTimeWindow with size 5 minutes. And we have events containing 2 basic pieces of information:
- number
- event timestamp
In this example, we kick off our Flink topology at 12:00 PM worker machines’ wall clock time (of course workers can have out of sync clocks but that’s out of the scope of this question). This topology contains one processing operator whose responsibility is to sum up the values of events belonging to each window and a KAFKA Sink which is irrelevant with regard to this question.
- This window has a BoundedOutOfOrdernessTimestampExtractor with allowed latency of one minute.
- Watermark: To my understanding, watermark in Flink and Spark Structured Stream is defined as (max-event-timestamp-seen-so-far - allowed-lateness). Any event whose event timestamp is less than or equal to this watermark will be discarded and ignored in result computations.
Part 1 (Determining Boundaries Of The Window)
Happy (Real-Time) Path
In this scenario several events arrive at the Flink Operator with different event timestamps spanning 12:01 - 12:09. Also, the event timestamps are relatively aligned with our processing time (shown in the X axis below). Since we're dealing with EVENT_TIME characteristic, whether or not an even belongs to a particular event should be determined via its event timestamp.
Old Data Rushing In
In that flow I have assumed the boundaries of the two tumbling windows are 12:00 -- 12:05 and 12:05 -- 12:10 just because we have kicked off the execution of the topology at 12:00. If that assumption is correct (I hope not), then what happens in case of a back-filling situation in which several old events coming in with much older event timestamps and we have kicked off the topology at 12:00 again? (old enough that our lateness allowance does not cover them). Something like the following:
- If it goes like that, then our events won't be captured in any window of course, so again, I'm hoping that's not the behavior :)
- The other option would be to determine windows' boundaries via the event timestamp of the arriving events. If that's the case, how would that work? The smallest event timestamp noticed becomes the beginning of the first window and from there based on the size (in this case 5 minutes), the consequent boundaries are determined? Cause that approach will have flaws and loopholes too. Can you please explain how does this work and how the boundaries of windows are determined?
Backfilling Events Rushing In
The answer to the previous question will address this as well, but I think it would be helpful to explicitly mention it here. Let's say I have this TumblingEventTimeWindow of size 5 minutes. Then at 12:00 I kick off a backfilling job which rushes in many events to the Flink operator whose timestamps cover the range 10:02 - 10:59; but since this is a backfilling job, the whole execution takes about 3 minutes to finish.
Will the job allocate 12 separate windows and populate them correctly based on the events' event timestamps? What would be the boundaries of those 12 windows? And will I end up with 12 output events each of which having the summed up value of each allocated window?
Part 2 (Unit/Integration Testing Of Such Stateful Operators)
I also have some concerns regarding automated testing of such logic and operators. Best way to manipulate processing time, trigger certain behaviors in such a way that shape desired windows' boundaries for testing purposes. Specially since the stuff that I've read so far on leveraging Test Harnesses seem a bit confusing and can cause some cluttered code potentially which is not that easy to read:
References
Most of what I've learned in this area and the source of some of my confusion can be found in the following places:
- Timestmap Extractors & Watermark Emitters
- Event Time Processing & Watermarking
- Handling Late Data & Watermarking in Spark
- The images in that section of Spark doc were super helpful and educative. But at the same time the way windows' boundaries are aligned with those processing times and not event timestamps, caused some confusion for me.
- Also, in that visualization, it seems like the watermark is computed once every 5 minutes since that's the sliding specification of the window. Is that the determining factor for how often the watermark should be computed? How does this work in Flink with regard to different windows (e.g.
Tumbling,Sliding,Sessionand more)?!
HUGE thanks in advance for your help and if you know about any better references with regard to these concepts and their internals working, please let me know.
UPDATES AFTER @snntrable Answer Below
If you run a Job with event time semantics, the processing time at the window operators is completely irrelevant
That is correct and I understand that part. Once you're dealing with EVENT_TIME characteristics, you're pretty much divorced from processing time in your semantics/logic. The reason I brought up the processing time was my confusion with regard to the following key question which still is a mystery to me:
How does the windows' boundaries are computed?!
Also, thanks a lot for clarifying the distinction between out-of-orderness and lateness. The code I was dealing with totally threw me off by having a misnomer (the constructor argument to a class inheriting from BoundedOutOfOrdernessTimestampExtractor was named maxLatency) :/
With that in mind, let me see if I can get this correct with regard to how watermark is computed and when an event will be discarded (or side-outputted):
- Out of Orderness Assigner
- current-watermark =
max-event-time-seen-so-far - max-out-of-orderness-allowed
- current-watermark =
- Allowed Lateness
- current-watermark =
max-event-time-seen-so-far - allowed-lateness
- current-watermark =
- Regular Flow
- current-watermark =
max-event-time-seen-so-far
- current-watermark =
And in any of these cases, whatever event whose event timestamp is less than or equal to the current-watermark, will be discarded (side-outputted), correct?!
And this brings up a new question. When would you wanna use out of orderness as opposed to lateness? Since the current watermark computation (mathematically) can be identical in these cases. And what happens when you use both (does that even make sense)?!
Back To Windows' Boundaries
This is still the main mystery to me. Given all the discussion above, let'e revisit the concrete example I provided and see how the windows' boundaries are determined here. Let's say we have the following scenario (events are in the shape of (value, timestamp)):
- Operator kicked off at 12:00 PM (that's the processing time)
- Events arriving at the operator in the following order
- (1, 8:29)
- (5, 8:26)
- (3, 9:48)
- (7, 9:46)
- We have a TumblingEventTimeWindow with size 5 minutes
- The window is applied to a
DataStreamwithBoundedOutOfOrdernessTimestampExtractorwhich has 2 minutemaxOutOfOrderness
- The window is applied to a
- Also, the window is configured with
allowedLatenessof 1 minute
NOTE: If you cannot have both out of orderness and lateness or does not make sense, please only consider the out of orderness in the example above.
Finally, can you please layout the windows which will have some events allocated to them and, please specify the boundaries of those windows (beginning and end timestamps of the window). I'm assuming the boundaries are determined by events' timestamps as well but it's a bit tricky to figure them out in concrete examples like this one.
Again, HUGE thanks in advance and truly appreciate your help :)
