how can I pass KafkaAvroSerializer into a Kafka ProducerRecord?

Question

I have messages which are being streamed to Kafka. I would like to convert the messages in avro binary format (means to encode them). I'm using the confluent platform. I have a Kafka ProducerRecord[String,String] which sends the messages to the Kafka topic. Can someone provide with a (short) example? Or recommend a website with examples?

Does anyone know how I can pass a instance of a KafkaAvroSerializer into the KafkaProducer? Can I use inside the ProducerRecord a Avro GenericRecord instance?

Kind regards

Nika

Hope this little example helps you kick it in: gist.github.com/mfirry/f65b16696e567b6edbede1ef9f924a8c — mfirry
Thanks! Do not I need a GenericRecord to store the values? Or is it enough if I integrate the KafkaAvroSerializer? — Nika

OneCricketeer OneCricketeer · Accepted Answer · 2019-07-19T04:40:06

You need to use the KafkaAvroSerializer in your producer config for the either serializer config, as well as set the schema registry url in the producer config as well (AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG)

That serializer will Avro-encode primitives and strings, but if you need complex objects, you could try adding Avro4s, for example. Otherwise, GenericRecord will work as well.

Java example is here - https://docs.confluent.io/current/schema-registry/serializer-formatter.html

how can I pass KafkaAvroSerializer into a Kafka ProducerRecord?

1 Answers