Windows Azure Inter-Role communication

2
votes

I want to create an Azure application which does the following:

  • User is presented with a MVC 4 website (web role) which shows a list of commands.
  • When the user selects a command, it is broadcast to all worker roles.
  • Worker roles process the task, store the results and notify web role
  • Web role displays the combined results of the worker roles

From what I've been reading there seem to be two ways of doing this: the Windows Azure Service Bus or using Queues. Each worker role also stores the results in the database.

The Service Bus seems more appropriate with its publish/subscribe model, so all worker roles would get the same command and roughly the same time. Queues seem easier to use though.

Can the service bus be used locally with the emulator when developing? I am using a free trial and cannot keep the application constantly whilst still developing. Also, when using queues how can you notify the web role that processing is complete?

3
azureasp.net-mvc-4queueazureservicebus
I realise this is now an old question, but I am curious why you have multiple work roles all responding to the same message? Are they each doing different tasks (based on that message)? – Gone Coding
I was trying to create an application that generates a large load on an existing web application. So I was using the worker roles to send simultaneous http requests, and the web role to to start the load test – Matt

3 Answers

5
votes

I agree. ServiceBus is a better choice for this messaging requirement. You could, with some effort, do the same with queues. But, you'll be writing a lot of code to implement things that the ServiceBus already gives you.

There is not a local emulator for ServiceBus like there is for the Azure Strorage service (queues/tables/blobs). However, you could still use the ServiceBus for messaging between roles while they are running locally in your development environment.

As for your last question about notifying the web role that processing is complete, there are a several ways to go here. Just a few thoughts (not exhaustive list)...

  1. Table storage where the web role can periodically check the status of the unit of work.
  2. Another ServiceBus Queue/topic for completed work.
  3. Internal endpoints. You'll have to have logic to know if it's just an update from worker role N or if it is indicating a completed unit of work for all worker roles.
3
votes

I agree with Rick's answer, but would also add the following things to think about:

If you choose the Service Bus Topic approach then as each worker role comes online it would need to generate a subscription to the topic. You'll need to think about subscription maintenance of when one of the workers has a failure and is recycled, or any number of reasons why a subscription may be out there.

Telling the web role that all the workers are complete is interesting. The options Rick provides are good ones, but you'll need to think about some things here. It means that the web role needs to know just how many workers are out there or some other mechanism to decide when all have reported done. You could have the situation of five worker roles receieving a message and start working, then one of them starts to repeatedly fail processing. The other four report their completion but now the web role is waiting on the fifth. How long do you wait for a reply? Can you continue? What if you just told the system to scale down and while the web role thinks there are 5 there is now only 4. These are things you'll need to to think about and they all depend on your requirements.

0
votes

Based on your question, you could use either queue service and get good results. But each of them are going to have different challenges to overcome as well as advantages.

Some advantages of service bus queues is that it provides blocking receipt with a persistent connection (up to 100 connections), it can monitor messages for completion, and it can send larger messages (256KB).

Some advantages of storage queues over the service bus solution is that it's slightly faster (if 15 ms matters to you), you can use a single storage system (since you'll probably be using Storage for blob and table services anyways), and simple auto-scaling. If you need to auto-scale your worker roles based on the load, passing the the requests through a storage queue makes auto-scaling trivial -- you just setup auto-scaling in the Azure Cloud Service UI under the scale tab.

A more in-depth comparison of the two azure queue services can be found here: http://msdn.microsoft.com/en-us/library/hh767287.aspx

Also, when using queues how can you notify the web role that processing is complete?

For the Azure Storage Queues solution, I've written a library that can help: https://github.com/brentrossen/AzureDistributedService. It provides a proxy layer that facilitates RPC style communication from web roles to worker roles and back through Storage Queues.