Getting lots of Task Canceled Exception when using Azure Search .Net SDK to merge or upload documents asynchronously

0
votes

We are using Azure Search .Net SDK(v3.0.4) to merge or upload documents to one of the azure search index.

We are using the below code for the same:

var batch = IndexBatch.MergeOrUpload(documentList);
await searchIndexClient.Documents.IndexAsync(batch);

We are indexing 2 types of subindexes using the same search client. When there is a high load, we are facing lots of task cancelled exceptions and the operation is failing without any hint to inner exceptions.

Stacktrace for reference: System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Rest.RetryDelegatingHandler.<>c__DisplayClass11_0.<b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Rest.RetryDelegatingHandler.d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Search.DocumentsOperations.d__23.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Search.DocumentsOperationsExtensions.d__13`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---

2
azureazure-cognitive-search
Which SDK are you using - I couldn't find that version? Can you post a link to the NuGet package? The latest versions can be found here: aka.ms/azsdk/releases – Jon Gallant
This looks like you're pushing the service beyond its limits and it's not able to keep up with the request rate. I'll follow up with you offline, since we'll need to look at service telemetry to figure out what's going on here. – Bruce Johnston

2 Answers

0
votes

You should post the exact exception, with the stack trace.

Have you tried to set a breakpoint in your code and look at what other information you have? It would also help to see a bit more of your code. How do you create your search client and what size are the batches for example.

We have seen similar issues where submitting a batch will randomly fail. In our case, I believe it was some intermittent problem with Azure. I wrapped all my calls to Azure Search services with retry functionality so that it will wait a bit and then retry for a number of times before it ultimately fails. It might be something similar that happens in your case if it's related to high load.

0
votes

Adding the solution that helped to benefit the community, from our offline discussions.

Issue could be due to starvation of TCP connection pool and adding the connection settings showed improvements. To resolve other exceptions while calling Search service (Task cancelled exceptions, Http Socket exceptions, IndexBatch exceptions, Null reference exceptions, Service too busy exceptions etc), additional modifications/fix were made as per the requirement:

  1. Increase the service replicas.
  2. Decrease load on Search service by reducing calls.
  3. Adding delays.

Thanks Bruce and Sonia.