Launch Azure Worker Role from Queue Trigger

Question

I have an Azure hosted MVC Web App where a user can request a report to be generated by pressing a button. This report is a compute intensive, long-running process that I only want to run at night. I have experience using Queue Triggered WebJobs to process background tasks; however, this job will require more resources than my Web App Service plan has and I don't want to run this compute process along side my Web App. My hope is that I can write a queue message for each request and then have something check that queue each night to see if it has any messages. If it does, create/start a new Worker Role instance of sufficient power/memory to handle the job, process the queue message(s), then shut down/and deallocate the worker to prevent ongoing charges.

I can't figure out the best way to check the queue before starting the Worker Role and only create/start the Worker if there is work to be done since it will be a largish instance I want to minimize uptime to keep costs down.

Answer 1

You can use create a triggered WebJob that uses a TimerTrigger that is set to wake up once a day at some early hour like 2:00AM. the method triggered by the TimerTrigger can then peek at the queue to see if a message exists. If one or more messages exist, kick off a worker role that actually dequeues and processes messages.

Answer 2

You could write a Web Job using a Queue Trigger so it's automatically triggered when a new message pops into the queue. Then you can host the Web Job in it's own App Service Plan, separate from the Web App, so it has it's own dedicated resources allocated.

Since you mention that you want to keep costs down, I would actually recommend instead you use an Azure Function. The Azure Function can be setup with a Queue Trigger as well, with the added benefit of only paying for your Azure Function when it is running using the "Consumption Plan" pricing option.

Here's a link that outlines how Azure Functions pricing works:

https://buildazure.com/2016/10/11/how-azure-functions-pricing-works/

Answer 3

Well, Rob's comment got me on to the Service Management library which then took me to Azure Automation and Runbooks. I think that will end up being the solution. I can create a Runbook using PowerShell cmdlets that will peek the storage queue and if it finds any messages it can create/deploy a new instance of the Worker Role, which will look at the queue on startup and start processing any messages. The tricky part looks to be shutting down the Worker Role. The Self-Destruct/suicidal cloud service examples I have found only seem to kill off multiple instances, but not the last instance. Since I will only have one instance running, I don't think I can have it kill itself and get it into a Unallocated state. I think the solution is to use Runbooks again. Have the Worker Role write a "finished" message to another queue and have a scheduled Runbook watch this queue every x minutes. If it finds a message, stop and deallocate the worker role. Given that Functions has a hard run time limit, I can't use that. And cloud services gives me more resource options (more fine-tuned VM types) than anything I could get on WebJobs/Web App Service plans.