My application uses Azure storage blobs in order to download files from remote storage to the local machine.
How it works:
- The machine getting a queue message with URL (signed URL) to Azure Storage blob.
- The application sending GET to download the file. It's done with
HttpClient. - Application logic continues from there.
This is how I downloading the file:
HttpRequestMessage msg = new HttpRequestMessage(HttpMethod.Get, signedUrl);
HttpClient.Timeout = TimeSpan.FromSeconds(120);
var response = await HttpClient.SendAsync(msg);
response.EnsureSuccessStatusCode();
I found that few times per day, I got exception:
System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Security.SslStreamInternal.g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask
1 task, Int32 min, Int32 initial) at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory1 buffer) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates1 shouldRetryResultPredicates, Func5 onRetryAsync, Int32 permittedRetryCount, IEnumerable1 sleepDurationsEnumerable, Func4 sleepDurationProvider, Boolean continueOnCapturedContext) at Polly.AsyncPolicy1.ExecuteAsync(Func3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
At this point, I must clarify that this behavior is happening to ~0.1% of the URLs passed in the system.
I was suspecting that the problem is with the signed-URL. I found in all cases the signed URL was correct - I just put it in the browser and it was working.
Now when I know that the problem is not the URL itself, I thought it can be a network problem. So, by using Polly I have added 5 retries with an exponential delay between each one. Up to 5 retries. Delay formula - 3^n (where n is the number of retries). In the worst case, it will wait 4 minutes before trying again... And even now - still same error. All 5 retries were used but the file was not downloaded.
What to do? What else I should check?
