kafka streams + how to expire entries in a state store asynchronously

Question

I have a kafka streams topology that reads from an input topic updates some state and determines if the state entry needs to remain in state store or can be deleted. If it can be deleted it will be removed else I've a punctuator that runs every 10s and expires items from the state store.

I recently found out that the punctuators run on the same stream thread and can potentially block processing of the stream.

What are some patterns I can use to execute the logic inside the punctuator in a separate thread pool to avoid blocking stream processing ?

Appreciate your help.

Answer 1

Matthias J. Sax already said, that's not possible with state stores, so far, so as he works at Confluent, I believe thats the latest news.

However, what we did in our case was using a KStream-KTable join instead of a state store. I'm not sure, if that's possible for your case, but let me explain, what we did, maybe it's of some use for you, as well:

We have two Topics A and B, Topic A is consumed with a KStream. Topic B is consumed with a KTable. We transform the KTable data, so we can join it on the KStream for Topic A. We join it, perform our operations and "delete" the data from Topic B by writing a null value with the original key to Topic B, using map and through. So when we get another record in Topic A, there are no longer values in our KTable to join with (exactly what we wanted).

I hope it helps.