How to send compacted data to Azure Stream Analytics?

Question

I have a system collecting data from a device and sending data in a JSON format to an Azure Eventhub. There is an Azure stream analytics stream, that takes the data from the eventhub and outputs it to PowerBI. It sends data every second. Sometimes, Internet connection is lost and the system keeps collecting data from the device. When the Internet connection comes back up, it sends all the collected data to eventhub. I would like it to zip or compress the data before sending.

But the Stream analytic input formats are only JSON,CVS or Avro. Would compressed Json be automatically un-compressed? Or we cannot send compressed data to Azure stream analytics?

Answer 1

Avro is compressed... kinda. The payload is compressed with Deflate. However, the message schema is included in uncompressed form. In some cases the schema for a single event could be larger than the payload. The more events of the same type that you send with each message, the more benefit you'll see from Avro.

If you're sending very small quantities of events and are concerned with payload size, you may be better suited rolling your own compression and using something like an Azure Function or WebJob to pull from a "raw" Event Hub, decompress then submit to a "decompressed" Event Hub that Stream Analytics can listen to.

Answer 2

Avro is a good way to compress (json) data when you do have to send a bunch of rows.

Regards,

Filip

Answer 3

Azure Stream Analytics does not support de-compressing input data.

Thanks, Kati Iceva