Azure Worker Role Scaling

1
votes

I am using an Azure Storage Queue and associated Worker Role to integrate a legacy system with a new system.

In brief, the legacy system sends application details to the queue, and the application details include a status such as "Pending", "Approved", "Cancelled" and "Completed".

The same application can be in the queue with different status (more than one message referring to same application).

If we scale up the Worker Role instance to 2, it is possible to process different statuses of the same application by two different worker roles. - Instance 1 process message with application status "Pending" - Instance 2 process message with application status "Approved"

I want to handle FIFO to messages with same application details even if there are many worker roles.

I am looking for an implementation method that will allow scaling of my Worker Roles whilst ensuring that I can sequentially process messages that relate to a given application.

2
azureazure-storage
Are you restricted to Azure Storage Queues or can you change to Azure Service Bus? If you can use Service Bus, you'd be able to have multiple receivers, each filtering on status. If the current status does not allow a move to the incoming status, you could assume that it is being processed elsewhere and requeue for receiving in a couple of minutes. – Brendan Green

2 Answers

1
votes

I found a good resource that solve the problem. Please follow the link Here.

0
votes

Don't put the data in the queue. Put it in the database. Only enqueue the information that you have to perform some work on a given application. The worker can the look into the database and see all the work for that application.

That way duplicate or out-of-order message delivery does not hurt. Duplicate delivery can happen anyway and you have to deal with it.