I am trying to write a Flink application which will read data from Kafka and increment a count in Hbase via a custom Hbase sink. I was looking into fault tolerance and I see that Flink will restart the job from the offset in the latest checkpoint. Doesnt this mean that my Hbase sink will produce duplicates? Do I have to implement my own dedupe logic?
0
votes
1 Answers
0
votes
Will HBase sink produce duplicates?
It depends on when you insert data into HBase.
How to achieve Exactly-Once semantics?
According to your description, what you need is end-to-end exactly-once semantics, which means that the data can be processed more than once but the insertion can only be executed once. Take a look at the CheckpointListener interface, and the method notifyCheckpointComplete is executed when the checkpoint is completed, which is a good time for you to commit data into HBase.
For more information, take a look at this document. https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
