Using Azure WebJob Timeout on Continuous Job with TimerTrigger

Question

I have a continuous WebJob with a function using the TimerTrigger to run a process every 30 seconds. A particular call in the function occasionally and seemingly randomly hangs, causing the webjob to wait indefinitely. Current solution is notice the service has stopped, then log into the Azure Dashboard and abort it manually.

Note that I know the correct course of action is to identify the root cause and fix it. Trust me, we're working on that. In the mean time, I want to treat the symptom, and need help doing so.

I'm attempting to have the WebJob detect if status using the Timeout decorator as described in this post on the Azure WebJobs SDK: https://github.com/Azure/azure-webjobs-sdk/issues/590. Implementing the suggestion, I'm able to see that when the problematic call hangs, the Timeout is detected, but the WebJob still doesn't die. What I doing wrong here that won't kill the function to allow subsequent invocations?

Program.cs

static void Main()
{
    var config = new JobHostConfiguration();
    config.UseTimers();
    config.FunctionTimeout = new TimeSpan(0, 15, 0);
    var host = new JobHost(config);

    Functions.Initialize();
    host.RunAndBlock();
}

Functions.cs

[Singleton]
[Timeout("00:05:00")]
public async static Task PeriodicProcess([TimerTrigger("00:00:30", RunOnStartup = true)] TimerInfo timer, CancellationToken cancelToken, TextWriter log)
{
    log.WriteLine("-- Processing Begin --");

    List<Emails> cases = GetEmailsAndWhatNot();
    foreach (Email e in Emails)
    {
        try
        {
            ProblematicFunction_SendEmail(e, log);
        }
        catch(Exception e)
        {
            // do stuff
        }
    }
    log.WriteLine("-- Processing End -- ");
}


public static void ProblematicFunction_SendEmail(Email e, TextWriter log)
{
    // send email
}

WebJob Output During Issues

-- Processing Begin --

Timeout value of 00:05:00 exceeded by function 'Functions.PeriodicProcess' (Id: '0f7438bd-baad-451f-95a6-9461f35bfb2d'). Initiating cancellation.

Despite the webjob initiating cancellation, the function doesn't die. Do I need to monitor the CancellationToken? How far down do I need to propogate asynchronous calling? What am I missing here that will actually abort the process?

Answer 1

As TimerTrigger states about TimerTrigger:

Singleton Locks

TimerTrigger uses the Singleton feature of the WebJobs SDK to ensure that only a single instance of your triggered function is running at any given time.

Scheduling

If your function execution takes longer than the timer interval, another execution won't be triggered until after the current invocation completes. The next execution is scheduled after the current execution completes.

Here is my test for this scenario, you could refer to it:

• Use CancellationToken.None and never propogate the cancellation token

  

  Note: The function PeriodicProcess would be time out after 30 s, but the Time-consuming job is still running, and after the long-running job has done, the Processing End log would be printed.

• Propogate the cancellation token

  

  Note: If we propogate the cancellation token, the Time-consuming job would be cancelled immediately.

Code snippet

[Timeout("00:00:30")]
[Singleton]
public async static Task PeriodicProcess([TimerTrigger("00:00:10", RunOnStartup = true)] TimerInfo timer, CancellationToken cancelToken, TextWriter log)
{
    log.WriteLine($"-- [{DateTime.Now.ToString()}] Processing Begin --");

    try
    {
        await longRunningJob(log, cancelToken);
    }
    catch (Exception e)
    {
        // do stuff
    }
    log.WriteLine($"-- [{DateTime.Now.ToString()}] Processing End -- ");
}


private async static Task longRunningJob(TextWriter log, CancellationToken cancelToken)
{
    log.WriteLine($"-- [{DateTime.Now.ToString()}] Begin Time-consuming jobs --");
    await Task.Delay(TimeSpan.FromMinutes(1), cancelToken);
    log.WriteLine($"-- [{DateTime.Now.ToString()}] Complete Time-consuming jobs --");
}