Kafka Streams Retry PAPI and dead letter

Question

I am trying to implement a retry logic within kafka streams processor topology in the event there was an exception producing to a sink topic.

1. I am using a custom ProductionExceptionHandler to be able to catch exception that happen on "producer.send" to the sink topic upon context.forward

2. What criteria should I use to be able resend the message to an alternate sink topic if there was an exception sending to original sink topic. Could this be deduced from type of exception in producer exception handler without compromising the transactional nature of the internal producer in Kafka streams.

3. If we decide to produce to a dead letter queue from production exception handler in some unrecoverable errors, could this be done within the context of "EOS" guarantee or it has to be a custom producer not known to the topology.

Answer 1

Kafka Streams has not built-in support for dead-letter-queue. Hence, you are "on your own" to implement it.

What criteria should I use to be able resend the message to an alternate sink topic if there was an exception sending to original sink topic.

Not sure what you mean by this? Can you elaborate?

Could this be deduced from type of exception in producer exception handler

Also not sure about this part.

without compromising the transactional nature of the internal producer in Kafka streams.

That is not possible. You have no access to the internal producer.

If we decide to produce to a dead letter queue from production exception handler in some unrecoverable errors, could this be done within the context of "EOS" guarantee or it has to be a custom producer not known to the topology.

You would need to maintain your own producer and thus it's out-of-scope for EOS.