Replace Azure Service Bus with blob storage

Question

I have been using Azure Service bus for a couple of years. I have an API in front, that publish messages to a service bus topic. On that topic, I have two subscription, with Azure functions consuming each subscription. The one writes the message to blob storage and the other saves the data in a database.

              /--- subscription 1 -> Function 1 -> database
API -> topic <
              \--- subscription 2 -> Function 2 -> blob storage

I have had a lot of problems with performance on Azure Service Bus and ended up upgrading to a premium service bus plan. It's running great, but it's very expensive, considering I'm only using it to save a message to two storage types.

I'm thinking about the possibility to switch to another architecture where my API stores all messages in blob storage. Then create a function that triggers on a new blob in blob storage and saves the result to the database.

API -> blob storage -> function -> database

This would be a much cheaper approach, but I'm a bit cautious implementing this approach. With service bus, I have multiple ways of optimizing the performance like upgrading to an even larger plan, consuming messages in batches and more. With the blob storage approach, I put all of my eggs in the basket of blob storage and the performance of that.

Anyone with ideas, similar experiences, or something to contribute?

Have you tried Event Hubs? You could run at max 32+32 Functions instances doing database and blob writing in parallel (one reader per partition, 32 partitions max). You can also consume batches in Event Hubs. — juunas
@juunas I did look at Event Hub, yes. When having to choose between event hub and service bus, I do believe that my needs lean towards the features of service bus like: bigger message, messages not events, and more. The price on Event Hub sure seems appealing. But with my experience with service bus, I'm afraid that I will need a dedicated event hub to ensure good performance. The price for this is not available online (which means it's expensive). Thank you for your input though. Will take another look. — ThomasArdal
@ThomasArdal Don't worry about the performance of event hub, you can send a lot through it. — Mikhail Shilkov
How many messages (per second) and what size are we talking about? — Peter Bons
@Mikhael That's what they are saying about Service Bus as well. But being in a shared environment, always introduce the risk of a noisy neighbour. I haven't played with Event Hub yet, so my concerns may be unfair. — ThomasArdal

Arunprabhu Arunprabhu · Accepted Answer · 2018-07-12T03:02:27

If your use case needs the data to be stored in blob and database, you can do it simply using Azure Function with Blob Trigger.

Instead of publishing the messages to Topic, make the API to save the messages in blob storage. Whenever a new message is stored in blob, the function gets triggered, there you can save the message to database.

I used the blob trigger in a different scenario. My API uploads a zip file to blob, I had a logic app with blob trigger. The logic app sends a message to Service Bus Queue whenever it is triggered. My worker role listens for the messages from Service Bus Queue and processes the zip file and store it in my DB (I sent the blob URL as the message body). I got it replaced by Azure Function with Blob trigger and it is working fine.

Replace Azure Service Bus with blob storage

1 Answers