How to freeze new version of AVRO schema of a topic so that non compliant message is rejected?

Question

With Java Client producer, the message can be fine tune to comply with the schema format before publish to topics.

With kafka rest proxy, how to reject messages if the message unable to be deserialize with kafka avro schema version? This is to prevent junks to be added by clients that not comply to schema. I see that version upgrade automatically with new schema to a topics. How to restrict the messages publish to the topic? It could be due to producer clients of all bugs.

I am searching the document and I am new to learn kafka. I know consumer can be smarter with offset but i want to clean up junks from a topics. Thanks.

Answer 1

There's no way to prevent this with Apache Kafka without putting some reverse proxy that denies the schema update request in front of the Registry, for example. And you'd have to deny access to anyone bypassing that proxy

Answer 2

If I understood your question correctly, you need to reject all events those are non compatible with your standard schema, if its correct then it would be helpful if you try AVRO schema registry compatibility checks.

How to freeze new version of AVRO schema of a topic so that non compliant message is rejected?

If you want to freeze a version of schema, then use FULL/FULL_TRANSITIVE compatibility type with all required fields, now you cannot delete any field because registry wont allow you to delete non-optional fields, and can only add optional fields. So that you can freeze on removal of existing fields, however optional still can added and this shouldn't be a impact on other non-relevant clients.

Hope it helps!