How to route Event Hub messages to different Azure functions based on their message type

1
votes

I have an Azure Event Hub over which I would like to send various types of messages. Each message should be handled by a separate Azure Function, based on their message type. What is the best way to accomplish this?

Actually, I could create some JSON container with a type and payload property and let one parent Azure Function dispatch all the messages payloads - based on their type - to other functions, but that feels a bit hacky.

This question basically asks the same - however it is answered how it can be done using the IoT Hub and message routing. In the Event Hub configuration I cannot find any setting to configure message routing though.

Or should I switch to an Azure Message Queue to get this functionality?

3
azureazure-functionsazure-eventhub
Can you provide some more details about your requirements, such as how many types of messages are there, messages must be consumed in the order, messages must be processed in the real time, messages are parts of the stream pipeline, etc. – Roman Kiss
@Roman Kiss: Around 10.000 messages per day (5 types of messages) must be sent from 5 different on-premise locations, they do not need to be processed in order or realtime (but preferably within a minute though, this is not a hard req.) – Wouter van Koppen

3 Answers

1
votes

I would use Azure Streaming Analytics to route it to the different Azure Functions. ASAs allow you to specify Event Hubs as a source and several sinks (one of which can be multiple Azure Functions). You can read more about setting up Azure Streaming Analytics services through the Azure Portal here. You'll need to set up the Event Hub as your source (docs). You'll also need to set up your sink (docs). You write some MS SQL-like code to route the messages to the various sinks. However, ASAs are costly relative to other services since you're paying for a fixed amount of compute.

I put some pseudo code below. You'll have to swap it out based on how you configure you're ASA using the information from the attached MS Documentation.

SELECT
    *
INTO
    [YourOutputAlias]
FROM
    [YourInputAlias]
HAVING
    [CONDITION]

SELECT
    *
INTO
    [YourAlternateOutputAlias]
FROM
    [YourInputAlias]
HAVING
    [CONDITION]
0
votes

Based on your additional info about the business requirements and assuming that the event size < 64KB (1MB in preview), the following screen snippet shows an example of your solution:

enter image description here

The concept of the above solution is based on the pushing a batch of the events to the Event Domain Endpoint of the AEG. The EventHub Trigger function has a responsibility for mapping each event message type in the batch to the domain topic before its publishing to the AEG.

Note, that using the Azure IoT Hub for ingestion of the events, the AEG can be directly integrated to the IoT Hub and each event message can be distributed in the loosely decoupled Pub/Sub manner. Besides that, for this business requirements can be used the B1 scale tier for IoT Hub ($10/month) comparing to the Basic Event Hubs ($11.16). The IoT Hub has built-in a message routing mechanism (with some limitations), but a recently new feature of the IoT/AEG integration such as publishing a device telemetry message is giving a good support in the serverless architecture.

0
votes

I ended up using Azure Durable Functions using the Fan Out/Fan In pattern.

In this approach, all events are handled by a single Orchestrator Function which in fact is a Durable Azure Function (F1). This deserializes incoming JSON to the correct DTO. Based on the content of the DTO, a corresponding activity function (F2) is invoked which processes it.

enter image description here