Azure Media Service : Retry failed after 6 tries. (The operation was canceled.)

Question

To learn about Azure Media Service I'm following Tutorial: Upload, encode, and stream videos with Media Services v3. But currently stuck as code given in this article UploadEncodeAndStreamFiles is throwing Retry failed after 6 tries. (The operation was canceled.) exception during file upload at await blob.UploadAsync(fileToUpload);.

private static async Task<Asset> CreateInputAssetAsync(
        IAzureMediaServicesClient client,
        string resourceGroupName,
        string accountName,
        string assetName,
        string fileToUpload)
    {
        // In this example, we are assuming that the asset name is unique.
        //
        // If you already have an asset with the desired name, use the Assets.Get method
        // to get the existing asset. In Media Services v3, the Get method on entities returns null 
        // if the entity doesn't exist (a case-insensitive check on the name).

        // Call Media Services API to create an Asset.
        // This method creates a container in storage for the Asset.
        // The files (blobs) associated with the asset will be stored in this container.
        Asset asset = await client.Assets.CreateOrUpdateAsync(resourceGroupName, accountName, assetName, new Asset());

        // Use Media Services API to get back a response that contains
        // SAS URL for the Asset container into which to upload blobs.
        // That is where you would specify read-write permissions 
        // and the exparation time for the SAS URL.
        var response = await client.Assets.ListContainerSasAsync(
            resourceGroupName,
            accountName,
            assetName,
            permissions: AssetContainerPermission.ReadWrite,
            expiryTime: DateTime.UtcNow.AddHours(4).ToUniversalTime());

        var sasUri = new Uri(response.AssetContainerSasUrls.First());

        // Use Storage API to get a reference to the Asset container
        // that was created by calling Asset's CreateOrUpdate method.  
        BlobContainerClient container = new BlobContainerClient(sasUri);
        BlobClient blob = container.GetBlobClient(Path.GetFileName(fileToUpload));

        // Use Strorage API to upload the file into the container in storage.
        await blob.UploadAsync(fileToUpload);

        return asset;
    }

Following is the detailed exception & exception trace:

Count = 6
Data: {System.Collections.ListDictionaryInternal}
HResult: -2146233088
HelpLink: null
InnerException: {"The operation was canceled."}
InnerExceptions: Count = 6
Message: "Retry failed after 6 tries. (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.) (The operation was canceled.)"
Source: "Azure.Core"
StackTrace: "   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAnd

DebuggerNotification(Task task)\r\n at Azure.Storage.Blobs.BlobRestClient.BlockBlob.d__0.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Threading.Tasks.ValueTask1.get_Result()\r\n at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable1.ConfiguredValueTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.Specialized.BlockBlobClient.d__26.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.Specialized.BlockBlobClient.<>c__DisplayClass48_0.<<GetPartitionedUploaderBehaviors>b__0>d.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Thro w()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.PartitionedUploader2.<UploadInternal>d__19.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.BlobClient.d__29.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.BlobClient.<UploadAsync>d__28.MoveNext()\r\n at System.Runtime.ExceptionSe rvices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.BlobClient.d__20.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.BlobClient.<UploadAsync>d__16.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Storage.Blobs.BlobClient.d__12.MoveNext()\r\n at System.Run time.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at UploadEncodeAndStreamFiles.Program.<CreateInputAssetAsync>d__7.MoveNext() in C:\\Study\\Test Source Code\\Video Portal\\media-services-v3-dotnet-tutorials-master\\AMSV3Tutorials\\UploadEncodeAndStreamFiles\\Program.cs:line 210\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at UploadEncodeAndStreamFiles.Program.d__4.MoveNext() in C:\Study\Test Source Code\Video Portal\media-services-v3-dotnet-tutorials-master\AMSV3Tutorials\UploadEncodeAndStreamFiles\Program.cs:line 82\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.Comp ilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.GetResult()\r\n at UploadEncodeAndStreamFiles.Program.d__3.MoveNext() in C:\Study\Test Source Code\Video Portal\media-services-v3-dotnet-tutorials-master\AMSV3Tutorials\UploadEncodeAndStreamFiles\Program.cs:line 34" TargetSite: {Void MoveNext()}

{"The operation was canceled."}
CancellationToken: IsCancellationRequested = true
Data: {System.Collections.ListDictionaryInternal}
HResult: -2146233029
HelpLink: null
InnerException: {"Error while copying content to a stream."}
Message: "The operation was canceled."
Source: "System.Net.Http"
StackTrace: "   at System.Net.Http.HttpConnection.<SendAsyncCore>d__53.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at System.Net.Http.HttpConnectionPool.<SendWithNtConnectionAuthAsync>d__48.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at System.Net.Http.HttpConnectionPool.<SendWithRetryAsync>d__47.MoveNext()\r\n   at System.Runtime.ExceptionServices.Exce

ptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()\r\n at System.Net.Http.HttpClient.d__71.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwa iter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n at Azure.Core.Pipeline.HttpClientTransport.d__7.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.RequestActivityPolicy.d__10.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.RequestActivityPolicy.d__9.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.Compiler Services.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.ResponseBodyPolicy.d__5.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.ResponseBodyPolicy.d__3.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.LoggingPolicy.d__8.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaite r.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.LoggingPolicy.d__7.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.d__1.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at Azure.Core.Pipeline.RetryPolicy.d__11.MoveNext()" TargetSite: {Void MoveNext()} Task: null

{"Error while copying content to a stream."}
Data: {System.Collections.ListDictionaryInternal}
HResult: -2146232800
HelpLink: null
InnerException: {"Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.."}
Message: "Error while copying content to a stream."
Source: "System.Net.Http"
StackTrace: "   at System.Net.Http.HttpContent.<CopyToAsyncCore>d__49.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Net.Http.HttpConnection.<SendRequestContentAsync>d__58.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Net.Http.HttpConnection.<SendAsyncCore>d__53.MoveNext()"
TargetSite: {Void MoveNext()}


{"Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.."}
Data: {System.Collections.ListDictionaryInternal}
HResult: -2146232800
HelpLink: null
InnerException: {"The I/O operation has been aborted because of either a thread exit or an application request."}
Message: "Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.."
Source: "System.Net.Sockets"
StackTrace: "   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\r\n   at System.Net.Security.SslStream.<<WriteSingleChunk>g__CompleteAsync|210_1>d`1.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Net.Security.SslStream.<WriteAsyncChunked>d__209`1.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Net.Security.SslStream.<WriteAsyncInternal>d__216`1.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.

HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Net.Http.HttpConnection.d__66.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.IO.Stream.d__30.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.RequestContent.StreamContent.d__6.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Azure.Core.Pipeline.HttpClientTransport.PipelineRequest.PipelineContentAdapter.< SerializeToStreamAsync>d__8.MoveNext()\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Net.Http.HttpContent.d__49.MoveNext()" TargetSite: {Void ThrowException(System.Net.Sockets.SocketError, System.Threading.CancellationToken)}

{"The I/O operation has been aborted because of either a thread exit or an application request."}
Data: {System.Collections.ListDictionaryInternal}
ErrorCode: 995
HResult: -2147467259
HelpLink: null
InnerException: null
Message: "The I/O operation has been aborted because of either a thread exit or an application request."
NativeErrorCode: 995
SocketErrorCode: OperationAborted
Source: null
StackTrace: null
TargetSite: null

Have anyone else faced such issue? If Yes, can you please help what might be causing this issue? Am I missing an permission causing this issue?

Answer 1

This seem to happen when you are on a low bandwidth upload network. You can add

BlobUploadOptions upOptions = new BlobUploadOptions
        {
            TransferOptions = new StorageTransferOptions
            {
                MaximumTransferSize = 4 * 1024 * 1024,
                InitialTransferSize = 4 * 1024 * 1024
            }
        };
        // Use Strorage API to upload the file into the container in storage.
        await blob.UploadAsync(file,upOptions);

that will change the chunk size and make it work with the lower bandwidth.

Answer 2

I agree with Jason's answer, just a couple of modifications.

First I would remove the endless while loop. Second, referencing the github issue thread, I think the most important point out that the BlobClientOptions should be passed when constructing either the BlobContainerClient or BlobClient. Then specify Transport. As mentioned here in the thread Tweaking Transport in BlobClientOptions

new BlobClientOptions
{
    Transport = new HttpClientTransport(
        new HttpClient { Timeout = TimeSpan.FromSeconds(102) })
};

I saw that the above in Jason's comment he left this commented out, but this would be an integral part solving the problem.

You can also look to extending the retry period this request requires more than 6 tries to make a successful connection.

new BlobClientOptions // or whatever options class suitable for your client
{
    Transport = new HttpClientTransport( new HttpClient { Timeout = Timeout.InfiniteTimeSpan } ),
    Retry = { NetworkTimeout = Timeout.InfiniteTimeSpan }
}

Answer 3

You need to change below code.

var sasUri = new Uri(response.AssetContainerSasUrls.First());
// Use Storage API to get a reference to the Asset container
// that was created by calling Asset's CreateOrUpdate method.  
BlobContainerClient container = new BlobContainerClient(sasUri);
BlobClient blob = container.GetBlobClient(Path.GetFileName(fileToUpload));
// Use Strorage API to upload the file into the container in storage.
await blob.UploadAsync(fileToUpload);
return asset;

Try to use my samle code, then tell me the test result, whether it solves your problem.

var sasUri = new Uri(response.AssetContainerSasUrls.First());
//var options = new BlobClientOptions
//{
//    Transport = new HttpClientTransport(
//        new HttpClient { Timeout = TimeSpan.FromSeconds(102) })
//};
var container = new Azure.Storage.Blobs.BlobContainerClient(sasUri);//,options);
int i = 0;
while (true)
{
    var blob = containerClient.GetBlobClient($"yourblobname");

    await using (var fileStream = System.IO.File.OpenRead(fileToUpload)
    {
        await blobClient.UploadAsync(fileStream, true, CancellationToken.None);
    }
}

For more details, please read this issue.

Azure.Storage.Blobs.BlobClient.UploadAsync throws System.Net.Sockets.SocketException #9212