Kafka ktable corrupt message handling

0
votes

We are using Ktabke for aggregation in kafka, its very basic use and have referred from kafka doc.

I am just trying to investigate that if some message consumption fails while aggregating how we can move such message to error topic or dlq.

I found something similar for KStream but not able to find for KTable and i was not able to simply extends KStream solution to ktable.

Reference for KStream Handling bad messages using Kafka's Streams API

My use case is very simple for any kind of exception just move to error topic and move on to different message

1
apache-kafkaapache-kafka-streams
Have you seen docs.confluent.io/current/streams/… ? – Michael G. Noll
Yeah i have read this, but this looks like only handles ser/deser error, what i am looking for is any kind of exception whether application related or kafka related. – Abhishek
Perhaps you can clarify the failure scenarios you are trying to cover when you say "I am just trying to investigate that if some message consumption fails". – Michael G. Noll
It could be any unforeseen issue like NPE in the application logic while aggregating, or lets say i am publishing the aggregated result to another kafka topic and that broker is down or it could be anything. If there a catch all kind of exception handling. Basically i want to achieve same result as we have in mq where if there is any kind of error it just put the message to error queue and move on to the next message. Basically i dont want my application to go fown if there is any corrupt message, i wany my application to move that corrupt message to error topic and move on to next message – Abhishek
"lets say i am publishing the aggregated result to another kafka topic and that broker is down or it could be anything." This type of errors is automatically handled by Kafka Streams. – Michael G. Noll

1 Answers

0
votes

There is no built in support for what you ask atm (Kafka 2.2), but you need to make sure that your application code does not throw any exceptions. All provided handlers that can be configured, are for exceptions thrown by Kafka Streams runtime. Those handlers are providing, because otherwise the user has no chance at all to react to those exception.

Feel free to create feature request Jira.