Asynchronous recurring task scheduler in Windows Azure

Question

We would like to make our customers able to schedule recurring tasks on a daily, weekly and monthly basis. Linear scalability is really important to us, that is why we use Windows Azure Table Storage instead of SQL Azure. The current design is the following: - Scheduling information is stored in a Table Storage table. For example: Task A, daily; Task B, weekly; ... - There are worker processes, which run hourly and query this table. Then decide, they have to run a given task or not.

But what if, multiple worker roles start to run the same task?

Some other requirements: - The worker processes can be in different time zones.

Windows Azure Queue Storage could solve all cuncurrency problems mentioned above, but it also introduces some new issues: - How many queue items should we generate? - What if the customer changes the recurrence rate or revokes the scheduling?

So, my question is: how to design a recurring task scheduler with multiple asynchronous workers using Windows Azure Storage?

Answer 1

Perhaps the new Azure Scheduler service could help?

http://www.windowsazure.com/en-us/services/scheduler/

Answer 2

Some thoughts:

But what if, multiple worker roles start to run the same task?

This could very well happen. To avoid this, what you could do is have a worker role instance (any worker role instance from the pool) read from table and push messages in a queue. While this instance is doing this work, all other instances wait. To decide which instance does this work, you can make use of blob lease functionality.

Some other requirements: - The worker processes can be in different time zones.

Not sure about this. Assuming you're talking about Cloud Services Worker Roles, they could be in different data centers but all of them will be in UTC time zone.

How many queue items should we generate?

It really depends on how much work needs to be done. You could put all messages in a queue. Only a maximum of 32 messages can be dequeued from a queue by a client at a time. So if you have say 100 tasks and thus 100 messages, each instance can only read up to 32 messages from the queue in a single call to queue service.

What if the customer changes the recurrence rate or revokes the scheduling?

That should be OK as once the task is completed you must remove the message from the queue. Next time when the task is invoked, you can read from the table again and it will give you latest information about the task from the table.

Answer 3

I would continue using the Azure Table Storage, but mark the process as "in progress" before a worker starts working on it. Since ATS supports concurrency which is controlled by Etags, you can be assured that two processes won't be able to start the same process

I would, however, think about retry logic when jobs fail unexpectedly and have a process that restarts job that appear to have gone orphan