Protobuf's bytes field in kafka streams

Question

I am trying to create a kafka-stream, in ksql cli, with following ksql command. As seen from the command, the source for the stream is kafka topic.

CREATE STREAM
TestStream
WITH
(
    kafka_topic = 'testtopic',
    value_format = 'PROTOBUF'
);

This kafka topic, uses schema-registry and has protobuf format. The schema is below.


message TestSchema {
  optional string deviceId = 1;
  required bytes  mac = 2;
  optional uint32 inNetwork = 3;
  optional sint32 rssi = 4;
  optional bool auth = 5;
}

At this point, when new message produced that kafka topic, I can't see the bytes mac field in the stream. Other fields, (deviceId, inNetwork, rssi and auth) shown as expected. I think protobuf's bytes field [doesn't supported] in the ksql.

Is it possible to make bytes supported by streams? Or, is it possible to transform bytes to string somehow in the stream creation ?

kszafran kszafran · Accepted Answer · 2021-05-04T16:11:18

I just into into this problem myself. Looks like bytes are not supported at the moment: https://github.com/confluentinc/ksql/issues/1742

Protobuf's bytes field in kafka streams

1 Answers