How to publish and consume nifi json to avro and avro to json content to kafka using NiFi

Question

I am new to both NiFi and Avro. So, according to my understanding if we use schema registry the schema won't be added to Avro content that is being published to Kafka, only schema ID will be sent is that correct??

How can I publish and consume through Kafka using Horton works Schema Registry, using Avro serialization and deserialization?

In Nifi ConvertJsonToAvro schema will be embedded while sending.SO, is there any other processor which will use schema registry and won't send schema while publishing.

Answer 1

On publishing side you would use PublishKafkaRecord (with the version corresponding to your Kafka broker) and you would configure it with a JsonTreeReader and an AvroRecordSetWriter. In the record writer you would configure the Schema Write Strategy as Hortonworks Content Encoded.

On consuming side your would ConsumeKafkaRecord (same version as publish) and you would configure it with an AvroRecordReader and a JsonRecordWriter. In the reader you would configure the Schema Access Strategy as Hortonworks Content Encoded.