Flink Kafka connector 0.10.0 Event time Clarification and ProcessFunction Clarification

2
votes

I'm struggling with an issue regarding event time of flink's kafka's consumer connector. Citing Flink doc

Since Apache Kafka 0.10+, Kafka’s messages can carry timestamps, indicating the time the event has occurred (see “event time” in Apache Flink) or the time when the message has been written to the Kafka broker.

The FlinkKafkaConsumer010 will emit records with the timestamp attached, if the time characteristic in Flink is set to TimeCharacteristic.EventTime (StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)).

The Kafka consumer does not emit watermarks.

Some questions and issues come to mind:

  1. How do I know if it timestamp taken is of the time it occurred or time written to the Kafka broker?

  2. If the consumer does not emit watermarks and TimeCharacteristic.EventTime is set, does this mean a message late by a few days can still enter and be processed?

  3. The main flow diagram does not contain a window function, and basically looks like the following: source(kafka)->filter->processFunction->Sink. Does this mean the the event is fired at the moment it is consumed by Kafka connector?

  4. I currently use Kafka connector 0.10.0, TimeCharacteristic.EventTime set and use a processFunction which every expectedly X minutes does some state cleanup. However I'm receiving a strange situation where the OnTimerContext contains timestamps which starts from 0 and grows until current timestamp when I start the flink program and is quite strange, is this a bug?

Thanks in advance to all helpers!

1
apache-flinkflink-streaming

1 Answers

3
votes
  1. That depends on the configuration of the Kafka producer that's creating these events. The message.timestamp.type property should be set to either CreateTime or LogAppendTime.
  2. Your flink application is responsible for creating watermarks; the kafka consumer will take care of the timestamps, but not the watermarks. It doesn't matter how late an event is, it will still enter your pipeline.
  3. Yes.
  4. It's not clear to me what part of this is strange.