Signal that an Azure worker role should not be scaled down while processing

Question

We have an Azure cloud service that has auto scaling enabled. We think that some of our messages occasionally are getting processed by a machine that is subsequently switched off by the auto scale down process. This means that the dequeue count on the message is increased and if this happens a couple of time we mark the message as failed when it is dequeued again (we let messages try to be processed twice)

Is the auto scaling choice of which machine to shut down purely based on the CPU usage of the worker role host process?

Is there a way to signal to the Azure infrastructure that a certain machine should not be turned off?

Answer 1

You can't refuse a shutdown within a role instance, but you can delay it, in OnStop() (I believe you can delay up to 5 minutes).

You'd probably want to prevent the role instance from ingesting more queue messages, while it was finishing its current item(s), which should be pretty straightforward (for example, setting a boolean in OnStop() that the queue-reading code checks, prior to getting another queue message).

Answer 2

You have designed your application with wrong expectations in mind. It's totally okay for any instance to disappear in the nowhere at any moment. Maybe it will be shut down maybe it just crashes. You application should be able to handle this. You need to logic for "abandoned" messages and "abandoning" should not be counted as usual failure. Maybe you want a count for it but at least that should be a separate counter.