Can we copy Azure blobs from one storage account to other storage accounts in parallel from same machine?

Question

In Microsoft Azure, I have a source storage account in one region and 3 destination storage accounts in 3 different regions. I want to copy blob data from source storage account to all 3 destination storage accounts. Currently I am using the azcopy (version 6) command in a bash script to do it. First it completes for one region then starts for another. It takes almost an hour everyday due to the geographical distance between the regions. I wanted to know if azcopy has any option to copy blobs from source to multiple destinations in a parallel manner. Any other suggestions to reduce the time are also invited :)

Generalization of azcopy command being used in my bash script:

/usr/bin/azcopy --source https://[srcaccount].blob.core.windows.net/[container]/[path/to/blob] --source-key $SOURCE_KEY --destination https://[destaccount].blob.core.windows.net/[container]/[path/to/blob] --dest-key $DEST_KEY --recursive --quiet --exclude-older

Answer 1

Azcopy can always only copy data from one source to one destination. But since you mention that you need to do this every day, I would probably go for a scheduled pipeline in Azure Data Factory instead. There you can also set up the three different copy jobs as parallel activities.

Answer 2

Just spawn a separate instance of your script for each destination. That way your copy will happen in parallel.

Here is a simple guide for doing this in BASH : https://www.slashroot.in/how-run-multiple-commands-parallel-linux