How to get all required Avro Schemas given a set of topics from the Confluent Schema Registry

1
votes

We're using Kafka, Avro and the Avro Schema Registry. Given a set of topics I want to consume, is there a way to get all schema IDs needed to decode the messages I'll receive?

I've checked the implementation of Confluent's Python client and what it seems to be doing is to receive messages, get the Avro schema ID from the individual message and then look up the schema from the Avro Schema Registry on the fly.

I'm looking for a way to get all schemas required before execution of the program (i.e. manually).

2
apache-kafkaavroconfluent-schema-registry
Is there a specific reason you need the schema ahead of time? Particularly when any given topic won't match the subject entry in the registry? The deserializer implementation, as you found, will lookup (and cache) the schemas as they're read. If you really wanted, you could extend that class to save the retrieved schemas to disk as their read. Saving all of them when they might not be needed seems wasteful - OneCricketeer

2 Answers

1
votes

Yes you can get the schema for any topic data

The rest api is 

GET /subjects/(string: subject)/versions

Get a list of versions registered under the specified subject. A subject refers to either a “-key” or “-value” depending on whether you are registering the key schema for that topic or the value schema

Once you get the versions of schema you can get the schema for each version using

GET /subjects/(string: subject)/versions/(versionId: version)/schema

Reference

https://docs.confluent.io/current/schema-registry/docs/api.html

0
votes

You can get schema definitions available in your schema registry by running the API calls to schema registry like:

curl http://localhost:8081/schemas/ids/3

where the last number in the URL is the schema # that you are interested in. If you have multiple types of messages in the broker, you can change the last # in the URL to get the different schema definitions for different message types.

For detail on the API calls, please refer to: https://docs.confluent.io/3.3.0/schema-registry/docs/api.html#schemas

This is the version 3.3 of the confluent platform. You can change it to current to get the current platform documentation.