how flink stream access dynamic mapping data to avoid io operation

Question

I have to process a streaming log, such as

{"id":1, "name":"alice"}

each one log need to get the family address by accessing a mapping db. however, the data in db is changing.

So can i read the db in period to avoid io operation by each one log.

Answer 1

It seems you could solve your problem by having a custom RichMapFunction where you implement RichMapFunction#open to get the state from the database (and storing it in some data structure) before starting to process events.

You can then launch an auxiliary thread from that function which, from time to time, fetches the most up-to-date information from the database and update the data structure. This doesn't need any locking if you can fit the dataset in memory twice as you can just perform an atomic swap between the two data structures.