Azure WebJob Service Bus Trigger Fails to Complete

Question

I have an Azure WebJob set to run continuously which uses the Service Bus triggers to read queues and topics. As the queues are processed, messages get sent into a topic and I have one topic to manage updates to cache.

Temporarily, I disabled the cache updates by simply returning from the function so that the message is cleared out of the topic.

public static void ProcessSearchUpdate([ServiceBusTrigger("events", "SearchClientUpdates", AccessRights.Listen)] BrokeredMessage message, System.IO.TextWriter log)
    {
        return;
    }

Most of the time, things go as planned and the messages simply are read and discarded. However, sometimes, I get the following exception which I'm assuming is something from the webjobs sdk trying to write logs.

As you can see, my method is not doing anything and should therefore not timeout.

When it fails, it prevents any trigger from executing which is quite annoying. It seems whatever is failing in the SDK has a 5 minute timeout and it prevents anything from running for 5 minutes, plus the restart time.

Microsoft.WindowsAzure.Storage.StorageException: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
 --- End of inner exception stack trace ---
 at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 77
 at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromStream(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 774
 at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromByteArray(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1162
 at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1285
 at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 114
 --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<SaveAndCloseAsync>d__6.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
 Request Information
 RequestID:
 RequestDate:
 StatusMessage:

Amor Amor · Accepted Answer · 2017-05-15T05:55:38

The exception related to the WebJob log system. Azure WebJob SDK will periodically flush the logs to blob storage. To solve the timeout issue, we could increase the timeout value of CloudBlobClient that WebJob SDK uses for this. Steps below are for your reference.

Step 1. Update the version of Microsoft.Azure.WebJobs to 2.0.0 using NuGet.

Step 2. Implement our own StorageClientFactory which could reset the timeout value of blob client.

public class MyStorageClientFactory : Microsoft.Azure.WebJobs.Host.StorageClientFactory
{
    public override CloudBlobClient CreateCloudBlobClient(StorageClientFactoryContext context)
    {
        CloudBlobClient client = context.Account.CreateCloudBlobClient();
        client.DefaultRequestOptions.ServerTimeout = TimeSpan.FromMinutes(60);
        return client;
    }
}

Step 3. Let JobHost use the custom StorageClientFactory.

JobHostConfiguration configuration = new JobHostConfiguration();
configuration.StorageClientFactory = new MyStorageClientFactory();

var host = new JobHost(configuration);

Azure WebJob Service Bus Trigger Fails to Complete

1 Answers