Azure Durable function - InvalidOperationException when CallActivityAsync

Question

I'm playing around with the Azure Durable functions. Currently I'm getting InvalidOperationException within Orchestration function after I call an activity. It complains that Multithreaded execution was detected. This can happen if the orchestrator function previously resumed from an unsupported async callback.

Have any one experienced such an issue? What I'm doing wrong? Complete code can be found on GitHub

Here is the line from the orchestration function:

var res = await ctx.CallActivityAsync<int>("LengthCheck", "inputData");

The LengthCheck activitiy function is:

[FunctionName("LengthCheck")]
public static Task<int> Calc([ActivityTrigger] string input)
{
    var task = Task.Delay(TimeSpan.FromSeconds(5));
    task.Wait();
    return Task.FromResult(input.Length);
}

The stack trace is:

ac6fd5cdd07a4dc9b2577657d65c4f27: Function 'InpaintOrchestration (Orchestrator)', version '' failed with an error. Reason: System.InvalidOperationException: Multithreaded execution was detected. This can happen if the orchestrator function previously resumed from an unsupported async callback.

at Microsoft.Azure.WebJobs.DurableOrchestrationContext.ThrowIfInvalidAccess()

at Microsoft.Azure.WebJobs.DurableOrchestrationContext.d__47`1.MoveNext()

End of stack trace from previous location where exception was thrown

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

Chris Gillum Chris Gillum · Accepted Answer · 2018-03-25T17:22:53

This exception happens whenever an orchestrator function does async work in an unsupported way. "Unsupported" in this context effectively means that await was used on a non-durable task (and "non-durable" means that it was a task that came from some API other than DurableOrchestrationContext).

You can find more information on the code constraints for orchestrator functions here: https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-checkpointing-and-replay#orchestrator-code-constraints.

Here are the rules that were broken in your code when I quickly scanned it:

Orchestrator code should be non-blocking. For example, that means no I/O and no calls to Thread.Sleep or equivalent APIs. If an orchestrator needs to delay, it can use the CreateTimer API.

Orchestrator code must never initiate any async operation except by using the DurableOrchestrationContext API. For example, no Task.Run, Task.Delay or HttpClient.SendAsync. The Durable Task Framework executes orchestrator code on a single thread and cannot interact with any other threads that could be scheduled by other async APIs.

This exception specifically occurs when we detect that an unsupported async call is made. I noticed that is happening in this code:

    private static async Task SaveImageLabToBlob(ZsImage imageLab, CloudBlobContainer container, string fileName)
    {
        var argbImage = imageLab
            .Clone()
            .FromLabToRgb()
            .FromRgbToArgb(Area2D.Create(0, 0, imageLab.Width, imageLab.Height));

        using (var bitmap = argbImage.FromArgbToBitmap())
        using (var outputStream = new MemoryStream())
        {
            // modify image
            bitmap.Save(outputStream, ImageFormat.Png);

            // save the result back
            outputStream.Position = 0;
            var resultImageBlob = container.GetBlockBlobReference(fileName);
            await resultImageBlob.UploadFromStreamAsync(outputStream);
        }
    }

The proper way to make async or blocking calls is to wrap them in activity functions, which don't have any of these constraints.

In more recent versions of this extension (v1.3.2 and greater), we've included a link to the documentation describing code-constraints in the exception message.

Azure Durable function - InvalidOperationException when CallActivityAsync

2 Answers