Get Avro record as binary array with kafka-avro-console-consumer

Question

I have a Kafka Producer which sends Avro records to a "test" topic. I also have a Schema Registry in which each record's schema is stored.

After that, I use the command

kafka-avro-console-consumer --topic test --zookeeper localhost:2181  --from-beginning

to retrieve the Avro records I sent, and it returns a JSON output like this:

{"A":"HI","B":"HELLO","C":"HEY","D":99999,"E":90,"F":"YO"}

The issue now, is that I want to get the output, not in JSON format, but as an array of Bytes, following the Avro binary format.

Something like for example:

[2, 65, 2, 78, 2, 78, -66, -102, 12, -76, 1, -16, 90, 0]

Is there any deserializer that I can use to accomplish this? I tried using

--value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer

and

-value-deserializer org.apache.kafka.common.serialization.ByteArrayDeserializer

but it did not work.

Any ideas?

Thank you.

OneCricketeer OneCricketeer · Accepted Answer · 2018-06-28T13:50:01

There is no builtin deserializer (actually, a Formatter) like that.

Avro console consumer uses a JSON formatter

Regular console consumer will output UTF8 strings for the bytes

You could build your own Formatter class, then add it to your Kafka Console Consumer classpath

--formatter <String: class>              The name of a class to use for
                                           formatting kafka messages for
                                           display. (default: kafka.tools.
                                           DefaultMessageFormatter)

Get Avro record as binary array with kafka-avro-console-consumer

2 Answers