Azure blob parallel copy from one storage to another

Question

I am currently learning about blob block storage. I would like to copy a block blob from one storage to another storage account.Usually when uploading from system to cloud the examples i saw was computing the block size and then using PutBlock and PutBlockList. I would like use the same method to copy from one storage account to another. using DownloadBlockList i am able to get blockid, but i am not able to get the data associated with block id.

CloudStorageAccount storageAccount = CloudStorageAccount.Parse( ConfigurationManager.AppSettings["StorageConnectionString"]);

            CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();

            CloudBlobContainer container = cloudBlobClient.GetContainerReference("input");

            CloudBlockBlob blob = container.GetBlockBlobReference("Cloud.mp4");

            List<string> commitedBlocks = new List<string>();

            IEnumerable<ListBlockItem> blockItem = blob.DownloadBlockList(BlockListingFilter.All);
            commitedBlocks.AddRange(blob.DownloadBlockList(BlockListingFilter.Committed).Select(id => id.Name));
            );

If i am able to get the data associated with the block id then i can do a parallel copy of the blocks.

Thanks

One clarification question: Do you want to copy a blob from one storage account to another or do you want to upload a file to multiple storage accounts in parallel? — Gaurav Mantri
what i am doing is more for my learning. I would like to copy from one storage account to another. If it is right would like to copy the blocks in parallel for faster copying — kumar

Gaurav Mantri Gaurav Mantri · Accepted Answer · 2015-07-14T07:58:27

If your objective is to copy blobs from one storage account to another, you don't have to do all of this :). Azure Storage API allows you to perform server side asynchronous copy operation. You simply send a request to Azure Storage Service to copy blob from one storage account to another and it will perform the copy operation. Since it's an asynchronous operation, you may want to track the operation status so that you will know when the copy operation has finished.

    private static void AsyncCopyExample()
    {
        var sourceAccount = new CloudStorageAccount(new StorageCredentials("source-account-name", "source-account-key"), true);
        var sourceContainer = sourceAccount.CreateCloudBlobClient().GetContainerReference("source-container-name");
        var sourceBlockBlob = sourceContainer.GetBlockBlobReference("source-blob-name");
        var targetAccount = new CloudStorageAccount(new StorageCredentials("target-account-name", "target-account-key"), true);
        var targetContainer = sourceAccount.CreateCloudBlobClient().GetContainerReference("target-container-name");
        var targetBlockBlob = sourceContainer.GetBlockBlobReference("source-blob-name");
        var copyId = targetBlockBlob.StartCopyFromBlob(sourceBlockBlob);//This will initiate the copy operation
        //Following code can be used to check if the copy has been completed.
        var isCopyOperationInProgress = true;
        do
        {
            targetBlockBlob.FetchAttributes();
            if (targetBlockBlob.CopyState.Status == CopyStatus.Pending)
            {
                Thread.Sleep(1000); //Sleep for a second and then check again
            }
            else
            {
                isCopyOperationInProgress = false;
            }
        } while (isCopyOperationInProgress);
    }

You may also find this blog post from Storage team useful: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/06/12/introducing-asynchronous-cross-account-copy-blob.aspx.

Azure blob parallel copy from one storage to another

1 Answers