Global Window allowedLateness

2
votes

In the gaming pipeline examples there is an example of a global/infite window that has an allowed lateness defined.

public PCollection<KV<String, Integer>> apply(PCollection<GameActionInfo> input) {
    return input.apply("LeaderboardUserGlobalWindow",
        Window.<GameActionInfo>into(new GlobalWindows())
            // Get periodic results every ten minutes.
            .triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()
                .plusDelayOf(TEN_MINUTES)))
            .accumulatingFiredPanes()
            **.withAllowedLateness(allowedLateness)**)
        // Extract and sum username/score pairs from the event data.
        .apply("ExtractUserScore", new ExtractAndSumScore("user"));
  }

Let us say the allowed Lateness is set to 36h.

Does this mean that there is an Window triggered every ten minutes than contains all the data from the beginning to those point minutes and each those of windows gets retriggered if any late data arrives?

E.g.

At after 30 minutes of processing time there would be windows:

A(10min)
B(20min)
C(30min)

If at minute 35 of processing time an message (X) arrives that has a event timestamp of minute 5, I would have the following outputs

old: A(10min)
old: B(20min)
old: C(30min)
new: A1(35min, equals all elements in A+X)

Is my understanding correct?

1
google-cloud-dataflow

1 Answers

0
votes

Yes. Your understanding is correct.

Also, any data later than 36hrs will be discarded.