confluent-kafka python avro messages

0
votes

So, I can successfully use the python confluent-kafka api to consume messages from Kafka that are text encoded.

When I try to consume Avro encoded (binary) messages, I get a UnicodeDecodeError exception on:

msg = kafka_consumer.poll(timeout=2.0)

Obviously, Avro encoded values are binary, not Unicode. With the Java API, I can specify a KafkaAvroDeserializer to the KafkaConsumer constructor. The Python API seems to have no parallel or similar configuration option to this.

The configuration options that can be passed to the underlying librdkafka don't seem to specify any custom (de)serializer or decoder or avro related options.

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

How do I consume Avro encoded messages with the Python confluent-kafka api?

1
pythonapache-kafkakafka-consumer-api

1 Answers

2
votes

On irc, #kafka, one of the developers said this is a bug in the 3.0 release of confluent-python, it's been fixed in the master branch, and the fix will be in the next point release.