Java Flink : Add Source as Stream and Sink in Batches

Question

I am new to Flink. And I have a requirement where in I need to read data continuously from a Kafka Stream but write it in Batches. So as to reduce the number of queries on MongoServer.

Please guide me the best way to do it.

What I tried to currently.

• Read data from Kafka Source
• Apply time window of 5 minutes
• Reduce the entries to create a list of entries.
• Read the list from the MongoSink function do a BulkWrite

Thanks, ashnik

Answer 1

The above seems like it should work. Since the Mongo client is pretty simple, if you wanted to be more efficient, you could implement your own stateful ProcessFunction that keeps a list of entries, and flushes to MongoDB when the list hits a certain size or sufficient time has elapsed.