Prevent Azure from Stopping Worker While Processing

Question

This may be kind of a simple question, but I haven't seen a direct answer yet. Say I'm using an Azure worker role to do some sort of long running task, say one that takes an hour. Now say MS decides that worker role needs some maintenance done on it and tries to shut it down 30 minutes into the work.

Is there a way to make Azure wait until that role finishes running to do maintenance? I do see the OnStop method, but it seems like you can only make things delay for a set amount of time before you get shut down anyway.

If this isn't possible, how do you plan around this for operations that take a decent amount of time and can't be split up into smaller chunks? Would you just rollback any changes made earlier and then retry the task?

Answer 1

You can handle the Stopping event to delay instance shutdown (in the graceful shutdown mode). In a crash situation, you can't really prevent shutdown...

Here's a suggestion for mitigation. Let's say your task is queue-driven. Let's pretend your message looks something like PROCESSWIDGET|123.

You read the queue message in a worker instance and start processing. You can actually modify the queue message along the way, and record your progress. So, in this fictitious example, let's say there are 4 steps to complete the hour-long task, each taking 15 minutes. As each step completes, you can update your message, appending a checkpoint status. Right after completing the first task, you update the message, and now it looks something like PROCESSWIDGET|123|STEP2 .

Now... something goes wrong and the VM instance dies for some reason. The queue message eventually becomes visible again, another worker instance reads it, and sees the message appended with the next step to perform. Assuming you've stored intermediate files in Blob storage (or some other persistent storage), you can pick up where you left off, and not re-process from the beginning.

See this MSDN page for details of the UpdateMessage() method.