flink window for time stamp

0
votes

I have stream of data like

Eventname, Event id, Start_time ( Time Stamp) ..

here I want to apply window transformation on last field Start_time which is of time stamp, my requirement is like I want to take data for last 30 minuts.

so what I see in flink window is something this .timeWindow(Time.minutes(30)) so I guess it take events of last 30 minutes but not respect to start_time

I want to take data where start_time is withing last 30 minuts, then how I write that transormation? do I need to use filter using that column ?

I am new to flink.

Thanks

1
scalaapache-kafkaapache-flinkflink-streaming

1 Answers

1
votes

There are two things you have to do:

  1. Enable event-time processing by calling setStreamTimeCharacteristic(TimeCharacteristic.EventTime) on the StreamExecutionEnvironment.
  2. Assign timestamps to your records and watermarks. This is done by using an AssignerWithPeriodicWatermarks or an AssignerWithPunctuatedWatermarks by calling `DataStream.assignTimestamps(yourAssigner).

In event-time mode, Flink will build the windows based on the timestamps that you assigned to your records. The watermarks tell Flink the "logical time of your data". A watermark of 1000 means that no more records with a timestamp less than 1000 are expected.

The whole topic of event-time processing is too complex to be discussed here. I'd recommend to have a look at the Apache Flink documentation.