copy azure storage containers to other account storage containers

1
votes

i am looking for a solution to copy containers with blobs from one azure storage to another azure storage, the easiest way. It needs to take the container name of the source storage and make an exact copy of the container name in the destination storage. Any help or ideas's? Is for a lot of containers 2500+.

2
azurecopystorageazure-storage-blobs

2 Answers

1
votes

AzCopy allows you to copy blobs within same storage accounts or between different storage accounts 

AzCopy.exe
 https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/
 https://<destaccount>.blob.core.windows.net/<destcontainer>/
 /sourcekey:<key> /destkey:<key> /S

The following optional parameters can be useful:

  • /MOV deletes source files after copying

  • /NC:<number> number of concurrent network calls (defaults to 8 x cores)

1
votes

Ok i got all the containers from storage to storage, this how i did it:

  1. I created a VM in azure in the same area where is my destination storage.

  2. I downloaded the command line tool "Cloudcopy" you can find it here: https://cloudcopy.codeplex.com/ the great thing about this tool is that i creates the containers for you in the case they don't exist in the destination storage (the -P in the end makes the container public by the way)

  3. I made a old school old fashion batch script (in my case all my containers are numbered 00000001 / 00000002 / etc.) that fetches all the containers to my VM temporary storage (d:) and after send the files to the destination storage and deletes the temporary storage (in case the temp storage is full, it's limited to 382GB in my case and my files exceed 2TB)

Here is the batch script i used (CMD):

@echo off
setlocal enabledelayedexpansion

REM Adds zeros to the container name
SET /a counter=100000000 
REM set the count starting value
SET /a count=1

:loop
REM Yes i know very old fashion loop

set /a calculation=%counter%+%count%
set directoryname=%calculation:~-8%
REM the ~-8% cuts off the 1 in the beginning from the counter

START /WAIT c:\\CloudCopy.exe "https://<STORAGENAMESOURCE>.blob.core.windows.net/%directoryname%/*.*" "d:\%directoryname%" "DefaultEndpointsProtocol=https;AccountName=<STORAGENAMESOURCE>;AccountKey=<STORAGEKEYSOURCE>"
START /WAIT c:\\CloudCopy.exe "d:\%directoryname%\*.*" "https://<STORAGENAMEDESTINATION>.blob.core.windows.net/%directoryname%" "DefaultEndpointsProtocol=https;AccountName=<STORAGENAMEDESTINATION>;AccountKey=<STORAGEKEYDESTINATION>" -P

rd /s /q d:\%directoryname%

set /a count=%count%+1

goto loop

Of course you can replace the %directoryname% variable with what ever you want, it works and you can ignore the whole counter part as example you can add a list of specific container names and for sure make a nicer loop. This one worked for me and i am quite busy so it's fine for me now :) Ah and by the way, Cloudcopy also supports very larges files, like in my case sizes of 2GB+ hench the VM in the destination area.