We planning to remove versions from kafka topic names. Currently schema version of the relevant message forms part of the topic name. But in future we will have large number of small variations of message and we don't want to create too many topics. But there are already many consumers for these topics. We don't want all those topics to make changes (should still try to consume from topic with version number). How can this be achieved? Are there any tools (e.g. AVRO) which can help achieve this? Has anyone experienced similar problem. And second question - how can consumer differentiate messages with small changes in structure coming to same topic.
1 Answers
1
votes
If I understood correct I would suggest:
- regarding to the first question:
pattern = Pattern.compile("topic_name_without_schema_version.*");
kafkaConsumer.subscribe(pattern, new ConsumerRebalanceListener());
- regarding to the second one: what kind of differentiation would you like to achieve? In case you just want to know if the current message is incompatible with the latest Avro Schema you can just try to convert current message and catch an Exception (if there is any), or you can generate an Avro schema based on the current message and check an equality of two schemas
