canceled while creating documents in cosmosdb

Question

We are using cosmosdb to log our application logs. Our logs have two one layer have general info and other have more detailed information of the general log. We also log a copy of these in other two collections for backup. So we have completely four collections. In our code we are using Async CreateDocumentAsync to create these logs. We are storing the Task in a list and using Task.WaitAll to complete all the tasks. In dev environment we did not see any errors but in production we are seeing canceled errors in our application insights when creating new documents. sample picture below.

Error info from application insights of API

Below is the code structure we are using

API Controller code

public void Log(LogModel logObject)
{
    try
    {
        _context.Save(logObject);
    }
    catch(Exception ex)
    {
        _azureTelemtry.TrackTrace($"Exception occured: {ex.ToString()}");
    }
}

Context save method

public void Save(LogModel logObject)
{
    List<Task> saveTasks = new List<Task>();
    LogInfo logInfo = logObject.logInfo;
    LogDetails logDetails = logObject.logDetails;
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogInfoCollectionID), logInfo));
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogInfoCollectionID), logInfo));    
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogDetailsCollectionID), logDetails));
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogDetailsCollectionID), logDetails));
    Task.WaitAll(saveTasks.ToArrays());
}

Exception Details

System.AggregateException: at MyApplication.Logging.Controllers.ApiController.Log (MyApplication.Logging, Version=2.0.0.0, Culture=neutral, PublicKeyToken=nullMyApplication.Logging, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: C:\TFS\CA\Logging\Release\Release 3.0-Hotfix\Logging\Controllers\ApiController.csMyApplication.Logging, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null: 57) at lambda_method (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null) at Microsoft.Extensions.Internal.ObjectMethodExecutor+<>c__DisplayClass33_0.b__0 (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__12.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__10.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+d__14.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__22.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__17.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+d__15.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Builder.RouterMiddleware+d__4.MoveNext (Microsoft.AspNetCore.Routing, Version=2.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware+d__11.MoveNext (Microsoft.AspNetCore.Server.IISIntegration, Version=2.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware+d__3.MoveNext (Microsoft.AspNetCore.Hosting, Version=2.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1+d__2.MoveNext (Microsoft.AspNetCore.Server.Kestrel.Core, Version=2.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) Inner exception System.Threading.Tasks.TaskCanceledException handled at MyApplication.Logging.Controllers.ApiController.Log:

Derviş Kayımbaşıoğlu Derviş Kayımbaşıoğlu · Accepted Answer · 2019-02-20T18:14:04

You do not await tasks. Besides Save function is synchronous thus Task.WaitAll(saveTasks.ToArrays()); runs synchronpusly.

At least I expect that public void Save(LogModel logObject) would be public Task Save(LogModel logObject) and Task.WaitAll(saveTasks.ToArrays()); would be return Task.WhenAll(saveTasks.ToArrays());

public Task Save(LogModel logObject)
{
    List<Task> saveTasks = new List<Task>();
    LogInfo logInfo = logObject.logInfo;
    LogDetails logDetails = logObject.logDetails;
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogInfoCollectionID), logInfo));
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogInfoCollectionID), logInfo));    
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, primaryLogDetailsCollectionID), logDetails));
    saveTasks.Add(documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(DatabaseId, archieveLogDetailsCollectionID), logDetails));

    return Task.WhenAll(saveTasks.ToArrays());
}

if your intention is to fire task and forget, now you can use _context.Save(logObject); but otherwise you need to use Task

another problem is: above code opens 4 tasks and process it uncontrollably. The default Concurrent Execution Limit in .Net Core is 10. Now think that Save method called three times one after another and none is finished yet. This means that you need to have 12 concurrent tasks. But .Net Core has limit to 10. Which I believe the last two tasks will be cancelled.

canceled while creating documents in cosmosdb

1 Answers