Pipe Events from Azure Event Hub to Azure Service Bus

Question

I am listening to Event hub for various events.

Each event is high value and cannot be missed.
Events are partitioned based on device id.
Events from one device id are sparse and not very frequent (couple of events per few dasy) . It only occurs in response to a user action which is infrequent.
The number of devices are huge , so I will have a lot of events for a variety of device Ids.

For each event , I need to make 3-4 API calls to Systems which are not super reliable. And since some of these are cross Geo Calls it might take some time.

I am planning to take the events from Event hub and put them into Service Bus. My reasons are as follows.

Event hub can be scaled to only 32 partitions and if one event takes time , the entire partition gets blocked.
Service bus on the other hand is more horizontally scalable. If the throughput drops I can just add more subscribers to the Service Bus.

I have been looking for patterns like this but I have not seen patterns where we get data from a log based messaging system and push them to a queue based one.

Are there better approach to handle such scenarios ?

Why not have the devices directly send to a Service Bus queue instead? — Serkant Karaca
The even hub is outside my control . Think of a company publishing their device data to event hubs. And they have a lot of consumers. Also Is Service bus optimized for such large number of devices keeping their connections ? — Abhik

Cindy Pau Cindy Pau · Accepted Answer · 2020-06-01T07:39:16

I think you can use Event hub trigger and service bus output binding to achieve what you want.

For example, I want to monitor Event hub 'test' and I am using C# library:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;

namespace FunctionApp68
{
    public static class Function1
    {
        [FunctionName("Function1")]
        [return: ServiceBus("test1", Connection = "ServiceBusConnection")]
        public static string Run([EventHubTrigger("test", Connection = "str")] EventData[] events, ILogger log)
        {
            var exceptions = new List<Exception>();
            string messageBodyt = "";
            foreach (EventData eventData in events)
            {
                try
                {
                    string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
                    messageBodyt = messageBodyt + messageBody;
                    // Replace these two lines with your processing logic.
                    log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
                    //await Task.Yield();
                }
                catch (Exception e)
                {
                    // We need to keep processing the rest of the batch - capture this exception and continue.
                    // Also, consider capturing details of the message that failed processing so it can be processed again later.
                    exceptions.Add(e);
                }
            }

            // Once processing of the batch is complete, if any messages in the batch failed processing throw an exception so that there is a record of the failure.

            if (exceptions.Count > 1)
                throw new AggregateException(exceptions);

            if (exceptions.Count == 1)
                throw exceptions.Single();
            return messageBodyt;
        }
    }
}

The above code will collect from event hub 'test' and save to service bus queue 'test1'.

Have a look of these doc:

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=csharp

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-output?tabs=csharp#example

Pipe Events from Azure Event Hub to Azure Service Bus

2 Answers