Confluent Schema Registry/Kafka Streams: prevent schema evolution

Question

Is there a way to configure Confluent Schema Registry and/or Kafka Streams to prevent schema evolution?

Motivation

We have multiple Kafka Streams jobs producing messages for the same topic. All of the jobs should send messages with the same schema, but due to misconfiguration of the jobs, it has happened that some of them send messages with fields missing. This has caused issues downstream and is something we want to prevent.

When this happens, we can see a schema evolution in the schema registry as expected.

Solution

We checked the documentation for Confluent Schema Registry and/or Kafka Streams, but couldn't find a way to prevent the schema evolution.

Hence, we consider to modify the Kafka Streams jobs to read the schema from Confluent Schema Registry before sending it. If the received schema matches the local schema of the messages, only then we send them.

Is this the right way to go or did we miss a better option?

Update: we found an article on medium for validating the schema against the schema registry before sending.

Answer 1

It depends which language and library you use and what kind of APIs do they provide. If you are publishing generic records, you can read and parse .avdl or .avsc file into the record type and build your event. Which means if event you are trying to build wouldn't be compatible with the current schema you won't be able even to build that event hence won't be able to modify existing schema. So in this case simply store with your source code a static schema. With specific record it is more or less the same, you can build your Java/C# or other language classes based on the schema, you build them then simply new them up and publish. Does it make any sense?) PS. I worked with C# libs for Kafka maybe some other languages do not have that support or have some other better options