How exactly does Azure stop Web Jobs?

Question

We're running continuous Web Jobs under a WebApp that is set to AlwaysOn.

One of the jobs, perhaps badly written, is set to run in a loop as long as there are items in an Azure Storage queue, and this queue is being populated faster than the job can consume - effectively making the loop infinite. I want to sometimes stop this job manually.

In the Azure Management Portal (manage.windowsazure.com), I've "Stopped" the job and restarted the WebApp, but the job still seems to be running. How exactly does Azure stop/kill a job?

Apparently it creates a 'disable.job' file in the job directory to signal not to runit, but that of course doesn't get checked until the next startup of the job, which never happens since it's in an infinite loop. Restarting the WebApp doesn't seem to help either.

The only thing that seems to help is when we redeploy which includes swapping with a staging environment (shutting down the previous one with the running job).

@Anton While it's similar, it isn't the same question. I asked how Azure stops jobs, but that question and responses from Amit, one of the Kudu developers, helped clarify. I'm posting my findings as an answer. — makhdumi

makhdumi makhdumi · Accepted Answer · 2015-10-30T16:39:43

The (partial) answer my question of how exactly Azure stops jobs can be found in the kudu source:

    private bool IsDisabled
    {
        get { return FileSystemHelpers.FileExists(_disableFilePath) || Settings.IsWebJobsStopped(); }
    }

So either if disable.job exists, or WEBJOBS_STOPPED is set in the application settings.

I already knew this though, and in our particular case it wasn't helping. It turned out the jobs were running on the swapped (and also stopped!!) staging environment. Jobs will keep running in a staging environment, even after the slot is stopped. To me, this is a serious bug with Azure, but no one seems to acknowledge it as an issue.

How exactly does Azure stop Web Jobs?

1 Answers