AzCopy a directory by Wildcard pattern Prefix

0
votes

Current Migration:

I'm currently in the process of migrating a storage account with 100TB + of data from one subscription to another. The legacy storage account has a very distinct folder structure:

  • ContainerName//{date}/filename.txt

Examples:

  • ContainerName//20140327/readme1.txt
  • ContainerName//20140328/readme2.txt

I don't know why they decided to use an empty folder directory // to house this data(before my time). There is a virtual directory on every date from 2014 up until present. Due to the sheer size of this migration, I would like to break it up into multiple azcopy jobs. For example migration 1 would migrate all data from 2014, migration 2 would migrate all data from 2015 and so on....

Problem:

Is there a way to perform an AzCopy based on a blob name prefix, since azure storage does not use real folders, rather it uses virtual folders. For example only migrate data with the following prefix: containerName//2014*

What I have tried:

I see that the most recent version of AzCopy now has --include-pattern command. I have tried multiple ways of doing the following:

azcopy.exe copy $srcContainerUrl $destContainerUrl --recursive --include-pattern "//2014*"

but it never seems to be able to find the files I am trying to migrate

1
azureazure-storagedata-migrationazcopy
Before you start, do two things: 1) Analyze the costs of moving 100 TB. 2) Perform a sample move to estimate time. Moving 100 TB will cost you around $10K and take months (~5 TB per week). That is just a rough guesstimate. I would talk to Azure support first. – John Hanley
Try using “—include-path” instead of “—include-pattern”. Based on the documentation, former is for virtual folders while latter is for blob names. However I don’t think using azcopy will solve your “//“ problem as the same structure will be copied in your new storage account. – Gaurav Mantri

1 Answers

0
votes

After going through your post, I recalled that there is a way through which you can migrate all the files and folders from one subscription to another according to the time preferences. You can use ‘—include-before’ or ‘--include-after’ option with the azcopy command. Here you have to specify date and time in ISO-8601 format. Below is an example and reference article from Microsoft: -

Syntax: - azcopy copy '*' 'https://..core.windows.net/' --include-after

Example: - azcopy copy 'C:\myDirectory*' 'https://mystorageaccount.blob.core.windows.net/mycontainer/FileDirectory' --include-after '2020-08-19T15:04:00Z'

Article: - https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs-upload?toc=/azure/storage/blobs/toc.json

P.S.: - Though, it is not the solution to your current problem, but the scenario as explained by you can be taken care of by this

Note: - Moving 100TB data from one subscription to another is not recommended as it takes more time (for months) and cost too.

Also, you can use the Azure Resource Mover to move the resource from one subscription to another. Please find the below reference links.

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/move-resource-group-and-subscription

https://docs.microsoft.com/en-us/azure/storage/common/storage-account-move?tabs=azure-portal

Thanking you,