This question was already asked here but since it's been two years I'm wondering if anything has changed.
I have a use-case in which I would like to share state between two Flink operators:
- Stream A is the main stream, it flows continuously
- Stream B is just a dataset of enrichment data. It's big (several GBs) and so will not fit in as a broadcast stream.
Stream B has an operator associated with it (FlatMap, but could be anything really) which acts as a state-loader and LOADS the enrichment data into RocksDB as a list-state.
- Then, I connect the streams, where I would like to have access to the same state that was created in the enrichment stream.
Lastly, I know I can simply load the entire state AFTER the streams have been connected using a "co" function. It's just that from a software engineering point-of-view, separating the responsibilities into a "state-loader" class and a actual "data-enricher" class seems cleaner and so I'd just like to know if it's possible.
Thank you.