Test/QA/Prod setup with a Kafka Schema Registry

Question

We are working with Kafka, and have three environments set up: A test environment for the test system, a QA environment and a production environment. Pretty standard. Now we are getting started with using Avro, and have set up a schema registry. The same installation setup: Test, QA and production. But now we get a bit uncertain about how to use the environments together. We have looked around a bit but haven't really found any examples of how to setup a Kafka test/qa/prod environment with a schema registry. These are the three approaches we have discussed internally:

1. Should we use Schema Registry Prod for all environments? Just as we do with our other artifact repositories.

With Nexus, Artifactory, Harbor etc. we use one instance for handling both developer version and release versions of artifacts. So our initial approach was to do the same with the Schema Registry. But here there is a difference though: with our other artifact repositories we have SNAPSHOT support and different spaces (builds/releases etc.) which we have not seen people using with a Schema Registry. So even though this approach was our initial, and it should work since we plan on using compatibility FULL_TRANSITIVE, we are now getting doubtful about sending each development/test version of a contract all the way to production. For example, FULL_TRANSITIVE would make it impossible to make non-compatible changes even during development.

2. Use the Test schema registry for test environment, and prod schema registry for Prod environment.

Straight forward, use "Kafka Schema Registry Test" in our test environment, with our Kafka test installation. And "Kafka Schema Registry Prod" only in our production environment. Then our build pipeline would deploy the Avro schemas to the production schema registry at an appropriate stage.

3. Use snapshots schemas

This would be to try and mimic the setup with our other repositories. That is, we use one schema registry in all our environments (the "Schema Registry Prod"). But during development and test we use a "snapshot" version of the schema (perhaps prepend "-snapshot" to the subject name). In the build pipeline we change to a 'non-snapshot' subject when ready to release.

So we would like to hear how other people work with Avro and Schema Registry. What does your setup look like?

Answer 1

Adding snapshot to the subject name would cause the default TopicNameStrategy serializer function to fail

In a primarily Java dev environment, we do have 3 Registries, but they are only at backwards transitivity, not full

Where possible, we use the Maven Plugins provided by Confluent to test the schema compatibility before registering. This will cause the CI pipeline to fail if the schema is not compatible, and we can use Maven profiles to override the registry url per environment, but the assumption is that if it fails in a lower environment, it'll never reach higher. Test environment can have SNAPSHOT artifacts manually be deployed to, but this is only referenced by code artifacts and not Registry subjects, so schemas need to be manually deleted if there are any mis-registrations. That being said, "staging/qa" and prod are generally the exact same, so barring network connectivity around "production", you'd only need two Registries

For non Java projects, we force them to use a standalone custom Maven archetype repo with the same plugins, so that allows them to version the schema artifacts in Maven, but also allows for JVM consumers to still use their data. It also simplifies the build pipeline support to a standard lifecycle