Route and transform data from Azure IoT Hub

1
votes

in our usecase, we receive messages on Azure IoT Hub, and would like to route the data to different Event Hubs or Service Bus topics.

IoT Hub routes and endpoints are no option, because the data is binary data (protobuf), and there are only 10 different endpoints possible (we need more).

Our requierements are:

  • Splitting the message
  • Transform the data (maybe json)
  • Routing to different endpoints based on the payload (different parts of message could be routed to different endpoints)
  • (optional) enrich the data with additional payload

I see different options:

  • Azure Stream Analytics
  • Azure Functions
  • Spark or Flink
  • Do it yourself (write an Application and run it in Service Fabric or Kubernets)

Which techology would you recommend?

Regards, Markus

1
azureapache-sparkapache-flinkazure-stream-analytics
Hi. At this time Azure Stream Analytics only supports JSON, CSV and AVRO, so it won't be able to read the incoming message if it's binary. If you can transform it before (e.g. with Azure function), you can process it further in Stream Analytics. – Jean-Sébastien
what is the data size? – Roman Kiss
@romanKiss the data size depends, because of very different messages. But in generell I would say, it's small. Maximum a few Kb, the most below 1 Kb. – mananana

1 Answers

1
votes

There is also another option for your scenario such as using an Azure Event Grid. In this case, the telemetry data from the Azure IoT Hub are pushed to the Event Grid via its custom topic endpoint. Note, that there is a limit for the event message such as 64KB, see more details here.

The Event Grid allows to subscribe unlimited number of the Event Hubs, more details about the Event Grid are here and here.

Based on the above, the following screen snippet shows your another option for routing a small telemetry data to more than 10 Event Hubs, basically to any kind of subscriber.

enter image description here