How to copy files from one container to another containers fits equally in all dest containers according to size using powershell

Question

I have one container in blob of storage account in azure contains different folder having files of different sizes. In other side, in same storage account, I have 10 different containers. I have to copy these files from source container to destination 10 containers but the condition is the files should be equally distributed to all the containers.

I have tried below so far

$key = "abcdxyz" 

# declaring the azure context

$context = New-AzStorageContext -StorageAccountName abcd  -StorageAccountKey $key 

#Getting the data from the blob

$bacdata = Get-AzStorageContainer -Name sourcecontainer*  -Context $context | Get-AzStorageBlob 

$15=$bacdata | where{$_.Name -like "sourcecontainer1*"} | where{$_.LastModified -gt (get-date).adddays(-1)}

I am beginner to powershell..firstly I am trying to just copy files from source container to destination container (not size wise) . Below is the code I tried to get the list of files I have to copy. — Sagar Kadam
$key = "abcdxyz" # declearing the azure context $context = New-AzStorageContext -StorageAccountName abcd ` -StorageAccountKey $key #Geting the data from the blob $bacdata = Get-AzStorageContainer -Name sourcecontainer* ` -Context $context | Get-AzStorageBlob $15=$bacdata | where{$_.Name -like "sourcecontainer1*"} | where{$_.LastModified -gt (get-date).adddays(-1)} — Sagar Kadam
Please add it to your question, not to the comment - for better visibility :) — Robert Dyjas

Mohit Verma Mohit Verma · Accepted Answer · 2019-06-10T08:34:44

Here is the powershell script to do it:

#Server side storage copy
$SourceStorageAccount = "sourceAccountName"
$SourceStorageKey = "sourceAccountAPIKey"
$DestStorageAccount = "destinationAccountName"
$DestStorageKey = "destinationAccountAPIKey"
$SourceStorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey

$Containers = Get-AzureStorageContainer -Context $SourceStorageContext

foreach($Container in $Containers)
{
    $ContainerName = $Container.Name
    if (!((Get-AzureStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName }))
    {   
        Write-Output "Creating new container $ContainerName"
        New-AzureStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
    }

    $Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $ContainerName
    $BlobCpyAry = @() #Create array of objects

    #Do the copy of everything
    foreach ($Blob in $Blobs)
    {
       $BlobName = $Blob.Name
       Write-Output "Copying $BlobName from $ContainerName"
       $BlobCopy = Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName
       $BlobCpyAry += $BlobCopy
    }

    #Check Status
    foreach ($BlobCopy in $BlobCpyAry)
    {
       #Could ignore all rest and just run $BlobCopy | Get-AzureStorageBlobCopyState but I prefer output with % copied
       $CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
       $Message = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100) 
       Write-Output $Message
    }
}

Overall concept would be similar , just change the function aas per the Az CLI. here is the copy command in Az CLI

az storage blob copy start

You can find more details here.

https://docs.microsoft.com/en-us/cli/azure/storage/blob/copy?view=azure-cli-latest#az-storage-blob-copy-start

Hope it helps.

How to copy files from one container to another containers fits equally in all dest containers according to size using powershell

1 Answers