TaskCanceledException when appending content concurrent using ConcurrentAppendAsync using CancellationToken.None

Question

I have written a log system that writes logentries in json format to a file in Azure Data Lake. Multiple threads are writing to that file so I use the ConcurrentAppendAsync method.

I initialize the access like this:

var clientCredential = new ClientCredential(ClientId, ClientSecret);
var creds = ApplicationTokenProvider.LoginSilentAsync(Domain, clientCredential).GetAwaiter().GetResult();
fileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);
fileSystem = fileSystemClient.FileSystem;

and I write the content like this:

    private async Task WriteToDataLakeAsync(IGrouping<string, LogEvent> logEvents)
    {
        var logEvent = logEvents.FirstOrDefault();
        if (logEvent == null)
            return;

        var filename = string.Format(@"Logs\{0:yyyy}\{0:MM}\{0:dd}\{1}\{2}.json", logEvent.Session.Timestamp, logEvent.Session.Origin, logEvent.Session.SessionId);

        var jsonBuilder = new StringBuilder();

        foreach (var @event in logEvents)
        {
            jsonBuilder.AppendLine(JsonConvert.SerializeObject(@event.SimplifyForBlob(), Formatting.None));
        }

        try
        {
            await fileSystem.ConcurrentAppendAsync(Account, filename, new MemoryStream(Encoding.UTF8.GetBytes(jsonBuilder.ToString())), AppendModeType.Autocreate, cancellationToken: CancellationToken.None);
        }
        catch (Exception ex)
        {
            telemetryClient.TrackException(new ExceptionTelemetry(new Exception($"{nameof(DataLake)}: Failed to write {filename}", ex)));
        }
    }

The problem is, although I pass CancellationToken.None to fileSystem.ConcurrentAppendAsync(...) I still get a TaskCanceledException in my catch block:

System.Exception: System.Threading.Tasks.TaskCanceledException: at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Rest.RetryDelegatingHandler+<>c__DisplayClass11_0+<b__1>d.MoveNext (Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Rest.RetryDelegatingHandler+d__11.MoveNext (Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Net.Http.HttpClient+d__58.MoveNext (System.Net.Http, Version=4.1.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Azure.Management.DataLake.Store.FileSystemOperations+d__10.MoveNext (Microsoft.Azure.Management.DataLake.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at Microsoft.Azure.Management.DataLake.Store.FileSystemOperationsExtensions+d__3.MoveNext (Microsoft.Azure.Management.DataLake.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089) at DHS.PlanCare.CentralLogging.EventHub.Processors.StreamProcessors.DataLake+d__15.MoveNext (DHS.PlanCare.CentralLogging.EventHub.Processors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\Sources\DHS\DHS.PlanCare.ManagementPortal\DHS.PlanCare.ManagementPortal.EventHub.Processors\StreamProcessors\DataLake.cs: 95)

(Line 95 is the call to ConcurrentAppendAsync)

What could be happening here? It does not help that the docs don't say anything about this exception. There is even a mistake in the documentation since it states that when a CancellationToken is not passed the value will be null, which is impossible since CancellationToken is a value type and therefor cannot be null.

So I do know for a fact that I do not initiate any kind of cancellation. The question is, can I safely ignore this exception or won't the data be written at all. Due to the large amount of data ingested and the fact that it happens on occasion it is hard to check if the data is actually written or not.

Not sure why this is voted to close for reasons of 'why is this code not working'. I really cannot think about any more I could append to this post. I have a specific problem and error but I do realize it cannot be reproduced easily but that why I was hoping to find people who did work with this technology and have dealt with it before. — Peter Bons
It's possible that the SDK client reached its timeout limit when waiting for a response from the server for the ConcurrentAppend call. You can configure this timeout when creating the DataLakeStoreFileSystemManagementClient. — Matt H
How much data are you transferring in a single ConcurrentAppend call, at the most? — Matt H
@MattH theoretically around 256kb as that is the max size of a batch message to an Azure Event Hub. But I expect it to be less per file, around 20kb. — Peter Bons

George Helyar George Helyar · Accepted Answer · 2017-10-12T07:22:54

HttpClient, which is used internally, throws an OperationCanceledException on timeout. These timeouts can happen during an active connection or when trying to establish a connection. DataLakeStoreFileSystemManagementClient has a constructor which takes an optional clientTimeoutInMinutes parameter, which is used to set HttpClient.Timeout.

In this case, it's actually the Microsoft.Rest.RetryDelegatingHandler responding to the cancellation that HttpClient has set which is throwing the exception.

It also commonly comes from an internal CancellationTokenSource.CancelAfter, which could also be used for a timeout.

As it's useful to differentiate between timeouts and real cancellation requests, I usually wrap in something like this:

try
{
    ...
}
catch (OperationCanceledException ex) when (!cancellationToken.IsCancellationRequested)
{
    throw new TimeoutException("The operation timed out.", ex);
}

TaskCanceledException when appending content concurrent using ConcurrentAppendAsync using CancellationToken.None

1 Answers