Azure BlobWriteStream The client could not finish the operation within specified timeout

Question

I am trying to upload 300GB (streamed) data to Azure blob storage. The code I am using to perform the upload looks like this:

var stream = blob.OpenWrite();
[...]
// the buffer is filled in with 128KB chunks of data from a larger 300GB file
stream.Write(buffer, offset, count);

After around 8h of uploading, I am receiving the following error message:

at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 77 at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.EndWrite(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\BlobWriteStream.cs:line 211

ErrorMessage = The client could not finish the operation within specified timeout.

As a side note, my upload speed is around 2MB/s (might be related to the timeout message). Any help would be appreciated.

What exactly are you trying to upload? Could you cut it in pieces? How many times have you tried it? A timeout for an 8+ hours attempt to upload can have many causes. — Juan Carlos Eduardo Romaina Ac
@JuanCarlosEduardoRomainaAc I am uploading a simple 300GB .txt file. I am reading the file in chunks and then sending each chunk to Azure via the BlobWriteStream. I have tried several times, and each time I receive the same timeout. — Mirel Vlad

Brando Zhang Brando Zhang · Accepted Answer · 2017-06-01T06:18:50

According to your description and error message, I suggest you could try to change the value of your BlobRequestOptions.MaximumExecutionTime to be longer if you don't want it to time out as quickly.

I suggest you could also enable the storage Diagnostics logs to look at your storage analytics logs and metrics to see if the latency is server or end to end latency. More details about monitor, diagnose, and troubleshoot Microsoft Azure Storage, you could refer to this article.

Besides, I suggest you could try to use Microsoft Azure Storage Data Movement Library to upload large files to blob storage.

This is designed for high-performance uploading, downloading and copying Azure Storage Blob and File. You could install it from the VS nuget package manager.

More details about how to use it, you could refer to this article.

Here is a example:

 CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
"connectstring");
            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
            CloudBlobContainer blobContainer = blobClient.GetContainerReference("foobar");
            blobContainer.CreateIfNotExists();
            string sourcePath = @"yourfilepath";
            CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference("foobar");

            TransferManager.Configurations.ParallelOperations = 64;

            SingleTransferContext context = new SingleTransferContext();
            context.ProgressHandler = new Progress<TransferStatus>((progress) =>
            {
                Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred);
            });

            var task = TransferManager.UploadAsync(
    sourcePath, destBlob, null, context, CancellationToken.None);
            task.Wait();

It will auto send each chunk to the azure storage as below:

Azure BlobWriteStream The client could not finish the operation within specified timeout

2 Answers