Uploading Large Files to Azure Blob Storage throwing Exception

Question

I have a scenario to upload databases backups to the Azure Blob storage via Window Service. It is working for the bak files size range between 300-500 MB but if the size exceeds 700 MB to 1 GB or more. It took more than an hour and then throw an exception.

Please check the code below let me know what I am doing wrong and what is the efficient method to upload large size files to blob storage. I have tried these two methods.

public static void UploadFile(AzureOperationHelper azureOperationHelper)
{
    CloudBlobContainer blobContainer = CreateCloudBlobContainer(tenantId, applicationId,
                                       clientSecret, azureOperationHelper.storageAccountName, azureOperationHelper.containerName,
                                       azureOperationHelper.storageEndPoint);
    blobContainer.CreateIfNotExists();


    var writeOptions = new BlobRequestOptions()
    {
        SingleBlobUploadThresholdInBytes = 50 * 1024 * 1024,//maximum for 64MB,32MB by default  
        ParallelOperationThreadCount = 12,
    };

    CloudBlockBlob blob = blobContainer.GetBlockBlobReference(azureOperationHelper.blobName);
    //blob.UploadFromFile(azureOperationHelper.srcPath);            
    blob.UploadFromFile(azureOperationHelper.srcPath, options: writeOptions);
}

public static void UploadFileStream(AzureOperationHelper azureOperationHelper)
{
    CloudBlobContainer blobContainer = CreateCloudBlobContainer(tenantId, applicationId,
                                       clientSecret, azureOperationHelper.storageAccountName, azureOperationHelper.containerName,
                                       azureOperationHelper.storageEndPoint);
    blobContainer.CreateIfNotExists();
    CloudBlockBlob blob = blobContainer.GetBlockBlobReference(azureOperationHelper.blobName);

    //byte[] contents = File.ReadAllBytes(azureOperationHelper.srcPath);
    //var writeOptions = new BlobRequestOptions()
    //{
    //    SingleBlobUploadThresholdInBytes = 50 * 1024 * 1024,//maximum for 64MB,32MB by default  
    //    ParallelOperationThreadCount = 12,        
    //};
    //blob.UploadFromByteArray(contents, 0, contents.Length, AccessCondition.GenerateIfNotExistsCondition(), options: writeOptions);
    blob.StreamWriteSizeInBytes = 100 * 1024 * 1024; //100 MB
    blob.UploadFromFile(string.Format(azureOperationHelper.srcPath));
    //using (var fs = new FileStream(azureOperationHelper.srcPath, FileMode.Open))
    //{
    //    blob.UploadFromStream(fs);
    //}            

}

Below are the exceptions I got.

Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden. at Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T](HttpStatusCode expectedStatusCode, HttpStatusCode actualStatusCode, T retVal, StorageCommandBase`1 cmd, Exception ex)

Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.

Why are you changing defaul block sizes? And what version of API/libraray are you using? — Matej
I am using Microsoft.WindowsAzure.Storage, Version=9.3.0.0. And I haven't change the default size in the first place but after getting the exception on large files I tried it by changing default block size but still, it didn't work. — Abdurrehman

Ivan Yang Ivan Yang · Accepted Answer · 2019-03-04T07:30:37

Please the code below, it works well at my side(about 2GB file, takes about 10 minutes for completing uploading):

public string UploadFile(string sourceFilePath)
{
    try
    {
        string storageAccountConnectionString = "AZURE_CONNECTION_STRING";
        CloudStorageAccount StorageAccount = CloudStorageAccount.Parse(storageAccountConnectionString);
        CloudBlobClient BlobClient = StorageAccount.CreateCloudBlobClient();
        CloudBlobContainer Container = BlobClient.GetContainerReference("container-name");
        Container.CreateIfNotExists();
        CloudBlockBlob blob = Container.GetBlockBlobReference( Path.GetFileName(sourceFilePath) );
        HashSet<string> blocklist = new HashSet<string>();

        byte[] fileContent = File.ReadAllBytes(sourceFilePath);
        const int pageSizeInBytes = 10485760;
        long prevLastByte = 0;
        long bytesRemain = fileContent.Length;

        do
        {
            long bytesToCopy = Math.Min(bytesRemain, pageSizeInBytes);
            byte[] bytesToSend = new byte[bytesToCopy];
            Array.Copy(fileContent, prevLastByte, bytesToSend, 0, bytesToCopy);
            prevLastByte += bytesToCopy;
            bytesRemain -= bytesToCopy;

            //create blockId
            string blockId = Guid.NewGuid().ToString();
            string base64BlockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId));

            blob.PutBlock(
                base64BlockId,
                new MemoryStream(bytesToSend, true),
                null
                );

            blocklist.Add(base64BlockId);

        } while (bytesRemain > 0);

        //post blocklist
        blob.PutBlockList(blocklist);

        return "Success";
    }
    catch (Exception ex)
    {
        return ex.Message;
    }
}

It works well for uploading large file(someone gives the solution from here).

Please let me know if you can work with it.

Uploading Large Files to Azure Blob Storage throwing Exception

1 Answers