Pre-validate messages sent to Kafka topic

Question

Is it possible to validate / filter messages that are being sent to Kafka topic?

Like, I want to ensure that only valid clients / producers send messages to my topic. I can certainly perform validation on the consumer side by discarding invalid messages based on certain parameters / criteria. But, what if I want to do it before the messages are written into the topic.

Say, Kafka receives a message, performs some validation and accordingly decides if it needs to discard or write that message into a topic. Is this possible?

Answer 1

A short answer - current versions of Kafka has no support for such functionality out of the box. And since Kafka producers are designed to communicate with multiple brokers during single session, there is no easy way to implement such ad-hoc filtering. There are couple of reasonable options still exists:

1. Use 2 topics: one "public" topic opened to everyone which will allow all messages, and another non-public "filtered" topic which will be populated by your own application with data from "public" after applying your filtering rules.
2. If you absolutely need to validate incoming messages before writing them down, then you could hide actual Kafka brokers behind some form of proxy application, which will do validation before writing messages into Kafka