Filter Kafka events and process them later

Question

I am using the Kafka Streams API (KTable, GlobalKTable ..). I am Consuming Kafka topic using KStreams. I have a requirement to filter out few incoming Kafka events based on some configuration and process them later when the configuration changes. Topics have a persistence limit of 7 days at least. Below is the requirement:

Key Value Status

K1 V1 Processed

K2 V2 Unprocessed (based on some business logic)

K3 V3 Processed

K4 V4 Processed

K1 V5 Processed ------>Current Offset<--------

Now I want to process the message (K2,V2) again. I was trying to make use of Ktables. But, not able to succeed. Since, I am relatively new to this concept, not sure KStream, KTable will be able to suffice this request.

Answer 1

It looks like you've got a problem where some messages are no 'processable' when first encounters, and you'd like to come back and process them at a later point in time.

The only solution to this that springs to mind would be for forward such messages to another topic for later processing, (the branch function may be of some use here), thereby allowing the processing of the main stream to continue in a linear fashion.

You'll need to use a custom Processor to process the delayed topic, which can choose to sleep for a period of time, or use some other logic to determine when to process a message.

However, such an approach may only be appropriate where unprocessed messages later become processable in the same order they were first encountered. If they're not, then you may run into issues where processable messages are sat behind unprocessable messages in your delayed queue. You may be able to get around this be having a timeout, after which a still unprocessable message is posted back to the end of the topic. But this all depends on your use-case.