I have a powershell azure runbook that iterates through a large storage account and enforces file age policies on the blobs within the account. This runs fine but runs up against the Fair Share policy of 3 hours. I can use hybrid workers but I would prefer to run multiple child runbooks in parallel each handling a different portion of the blob account using the first letter prefix.
Example:
First child runbook runs A-M Second: N-Z Third: a-m Fourth: m-z
I'm thinking of using a prefix variable within a loop that will iterate between letters.
## Declaring the variables
$number_of_days_bak_threshold = 15
$number_of_days_trn_threshold = 2
$current_date = get-date
$date_before_blobs_to_be_deleted_bak = $current_date.AddDays(-$number_of_days_bak_threshold)
$date_before_blobs_to_be_deleted_trn = $current_date.AddDays(-$number_of_days_trn_threshold)
# Number of blobs deleted
$blob_count_deleted = 0
# Storage account details
$storage_account_name = <Account Name>
$storage_account_key = <Account Key>
$container = <Container>
## Creating Storage context for Source, destination and log storage accounts
$context = New-AzureStorageContext -StorageAccountName $storage_account_name -StorageAccountKey $storage_account_key
$blob_list = Get-AzureStorageBlob -Context $context -Container $container
## Creating log file
$log_file = "log-"+(get-date).ToString().Replace('/','-').Replace(' ','-').Replace(':','-') + ".txt"
$local_log_file_path = $env:temp + "\" + "log-"+(get-date).ToString().Replace('/','-').Replace(' ','-').Replace(':','-') + ".txt"
write-host "Log file saved as: " $local_log_file_path -ForegroundColor Green
## Iterate through each blob
foreach($blob_iterator in $blob_list){
$blob_date = [datetime]$blob_iterator.LastModified.UtcDateTime
# Check if the blob's last modified date is less than the threshold date for deletion for trn files:
if($blob_iterator.Name -Match ".trn") {
if($blob_date -le $date_before_blobs_to_be_deleted_trn) {
Write-Output "-----------------------------------" | Out-File $local_log_file_path -Append
write-output "Purging blob from Storage: " $blob_iterator.name | Out-File $local_log_file_path -Append
write-output " " | Out-File $local_log_file_path -Append
write-output "Last Modified Date of the Blob: " $blob_date | Out-File $local_log_file_path -Append
Write-Output "-----------------------------------" | Out-File $local_log_file_path -Append
# Cmdle to delete the blob
Remove-AzureStorageBlob -Container $container -Blob $blob_iterator.Name -Context $context
$blob_count_deleted += 1
Write-Output "Deleted "$extn
}
}
Elseif($blob_iterator.Name -Match ".bak") {
if($blob_date -le $date_before_blobs_to_be_deleted_bak) {
Write-Output "-----------------------------------" | Out-File $local_log_file_path -Append
write-output "Purging blob from Storage: " $blob_iterator.name | Out-File $local_log_file_path -Append
write-output " " | Out-File $local_log_file_path -Append
write-output "Last Modified Date of the Blob: " $blob_date | Out-File $local_log_file_path -Append
Write-Output "-----------------------------------" | Out-File $local_log_file_path -Append
# Cmdle to delete the blob
Remove-AzureStorageBlob -Container $container -Blob $blob_iterator.Name -Context $context
$blob_count_deleted += 1
Write-Output "Deleted "$extn
}
}
Else{
Write-Error "Unable to determine file type." $blob_iterator.Name
}
}
Write-Output "Blobs deleted: " $blob_count_deleted | Out-File $local_log_file_path -Append
I expect to be able to run through the account in parallel.
