I am new to flink. I have two DataStream generated by FlatMapFunction, and I want to connect them by same key, what is one steam's element cant find matched key in other stream, what will flink do to this element , will it store in state forever? every flatmap will generate a bulk of elements, and I want to finish join on every flatmap.
what is the normal way to join two stream that maybe have unmatched element? thanks
Flink's online training site has a tutorial on the topic of expiring state that you might find helpful. There the approach is to use a ProcessFunction to implement the join by hand. In this example a timer is used to clear unmatched state. An alternative would be to use Flink's State TTL mechanism.
Some of Flink's built-in join functions also have a notion of narrowing the scope of the join so as to avoid indefinite state retention -- see the docs on joining for more details.
You can also use Flink SQL to do joins. You can either bound these joins explicitly in the query so they don't retain state indefinitely, or configure the SQL engine to expire state for you.
