Durable Function Fan out with Time limit - remains in "running" status

Question

I am attempting to implement a timeout for my Durable function.

In my function, I am currently doing a fan out of activity functions, each of which call a separate API to collect current pricing data. (Price comparison site). All of this works well and I am happy with the results, however I need to implement a time out in case 1 or more APIs do not respond within a reasonable time (~15 seconds)

I am using the following pattern:


 var parallelActivities = new List<Task<T>>
{
    context.CallActivityAsync<T>( "CallApi1", input ),
    context.CallActivityAsync<T>( "CallApi2", input ),
    context.CallActivityAsync<T>( "CallApi3", input ),
    context.CallActivityAsync<T>( "CallApi4", input ),
    context.CallActivityAsync<T>( "CallApi5", input ),
    context.CallActivityAsync<T>( "CallApi16", input )
};

var timeout = TimeSpan.FromSeconds(15);
var deadline = context.CurrentUtcDateTime.Add(timeout);

using ( var cts = new CancellationTokenSource() )
{
    var timeoutTask = context.CreateTimer(deadline, cts.Token);
    
    var taskRaceWinner = await Task.WhenAny(Task.WhenAll( parallelActivities ), timeoutTask);

    if ( taskRaceWinner != timeoutTask )
    {
        cts.Cancel();
    }
    
    foreach ( var completedParallelActivity in parallelActivities.Where( task => task.Status == TaskStatus.RanToCompletion ) )
    {
        //Process results here
    }
    //More logic here
}

Everything seems to work correctly. If any activity doesn't return within the time limit, the timeout task wins, and the data is processed and returned correctly.

The Durable functions documentation indicates that the Orchestrator states "This mechanism does not actually terminate in-progress activity function execution. Rather, it simply allows the orchestrator function to ignore the result and move on. For more information, see the Timers documentation."

Unfortunately my function remains in the "running" status until it ultimately hits the durable function timeout and recycles.

Am I doing something wrong? I realize that, generally, the durable function will be marked as running until all activities have completed, however the documentation above indicates that I should be able to "ignore" the activities that are running too long.

I could implement a timeout in each individual API, however that doesn't seem like good design and I have been resisting. So, please help me stackoverflow!

krishg krishg · Accepted Answer · 2020-07-06T17:46:26

According to this, The Durable Task Framework will not change an orchestration's status to "completed" until all outstanding tasks are completed or canceled even though output of those are ignored. Also, according to this and this, we can't cancel activity/sub-orchestration from parent at this moment. So, currently only way I can think of is to pass a Timeout param (of TimeSpan type) from parent as part of the input object to the activity (e.g. context.CallActivityAsync<T>( "CallApi1", input )) and let the child activity function handle it's exit respecting that timeout. I tested this myself and works fine. Please feel free to reach me for any follow up.

Durable Function Fan out with Time limit - remains in "running" status

1 Answers