Altering file names of Azure blobs in bulk

2
votes

How does one add a prefix to Azure blob files names in bulk?

I'm asking for an efficient way that can manage this operation for ~10M small sized blob files on Azure storage. My preference is doing it via Python (if possible).

If this isn't possible with the API currently, what alternatives can I resort to?

I need to download a selection of blob files from my Azure storage using azcopy (the only differentiating filter it gives me is prefix-based filtration). I need a workaround for processing small-sized millions of file objects. I'm also open to bulk deletion of blob files based on filename (and leaving behind ones I need) if push came to shove.

1
azureazure-blob-storage

1 Answers

3
votes

How does one add a prefix to Azure blob files names in bulk?

Adding a prefix is essentially renaming the blob which is not currently supported by Storage API. Furthermore each blob operation is atomic in the sense that it works on a single blob.

To rename the blob, you will need to perform 2 operations - Copy and Delete. What you will do is first copy the blob to another blob. The name of this new blob will be the name you want to give i.e. prefix + original blob name. Since you're copying the blob in the same blob container, it will be instantaneous. Once the blob is copied, you will then delete the original blob.

Edit: Bulk Deletion

Regarding your comment about bulk deletion, unfortunately there's no API to do so. You will need to delete each blob individually. You could possibly speed up this operation if you want to rename all blobs in a container. In that case, you will simply copy the blobs from source container to another container and once the blobs are copied, you can simply delete the source container. This obviously will not work if you only want to rename a subset of blobs in the blob container.