Reprocess enriched kstream data on insert/updates of KTable

Question

Here is a hypothetical but analogous scenario to what I am trying to achieve using Kafka Streams.

I have streaming data, sales, that I want to enrich with infrequently changing lookup data, say users & items, which I am planning to create KTable for. I plan to push this enriched data to a topic and to search engine using connect sink.

How do I ensure that updates in user/item data triggers enrichments for past sales data as well, not only the new data that is ingested in stream. As I understand, KTable inserts/updates doesn't trigger any reprocessing of past data of the stream.

I believe this may be a common use-case, at least I may not be the first one to have such need. Any guidance for solution or workarounds?

Answer 1

If you want to update old data, it implies that you want to do a table-table join. Note though, that for this case, all data of both inputs need to be hold on the application.