generate a Zip file from azure blob storage files

Question

I have some files stored in my windows azure blob storage. IWant to take these files and create a zip file and store to a new folder .Then return the path to the zip file. Set permission to the zip file location so that my users can download the zip file to their local machines by clicking on the link

 https://mystorage.blob.core.windows.net/myfiles/2b5f8ea6-3dc2-4b77-abfe-4da832e02556/AppList/isjirleq/mydocs1.doc
 https://mystorage.blob.core.windows.net/myfiles/2b5f8ea6-3dc2-4b77-abfe-4da832e02556/tempo/xyz/mymusic.mp3
 https://mystorage.blob.core.windows.net/myfiles/2b5f8ea6-3dc2-4b77-abfe-4da832e02556/general/video/myVideo.wmv
 https://mystorage.blob.core.windows.net/myfiles/2b5f8ea6-3dc2-4b77-abfe-4da832e02556/photo/photo1.png

I want to be able to loop therough these files and zip them all together to create a new zip file

(https://mystorage.blob.core.windows.net/myzippedfiles/allmyFiles.zip ) and return the path to the zip file

I have large number of files in my azure blob. So downloading them and zipping them and uploading is not a good idea.

How can I do this .I need some sample code to do this

Answer 1

We have solved this problem (partially) by zipping the files directly to the output stream using the blob streams. This avoids the issue of downloading zipping then sending and avoids the delay while this happens (we used ICSharpZipLib, reference). But it still means routing the stream through the web server:

  public void ZipFilesToResponse(HttpResponseBase response, IEnumerable<Asset> files, string zipFileName)
    {
        using (var zipOutputStream = new ZipOutputStream(response.OutputStream))
        {
            zipOutputStream.SetLevel(0); // 0 - store only to 9 - means best compression
            response.BufferOutput = false;
            response.AddHeader("Content-Disposition", "attachment; filename=" + zipFileName);
            response.ContentType = "application/octet-stream";

            foreach (var file in files)
            {
                var entry = new ZipEntry(file.FilenameSlug())
                {
                    DateTime = DateTime.Now,
                    Size = file.Filesize
                };
                zipOutputStream.PutNextEntry(entry);
                storageService.ReadToStream(file, zipOutputStream);
                response.Flush();
                if (!response.IsClientConnected)
                {
                   break;
                }
            }
            zipOutputStream.Finish();
            zipOutputStream.Close();
        }
        response.End();
    }

The storage service simply does this:

public void ReadToStream(IFileIdentifier file, Stream stream, StorageType storageType = StorageType.Stored, ITenant overrideTenant = null)
    {
        var reference = GetBlobReference(file, storageType, overrideTenant);
        reference.DownloadToStream(stream);
    }
private CloudBlockBlob GetBlobReference(IFileIdentifier file, StorageType storageType = StorageType.Stored, ITenant overrideTenant = null)
        {
            var filepath = GetFilePath(file, storageType);
            var container = GetTenantContainer(overrideTenant);
            return container.GetBlockBlobReference(filepath);
        }

Answer 2

Since blob storage is "just" an object store, you would need to download them somewhere (it could be a web/worker role or your local computer), zip them and then reupload the zip file. That's the only way to do it as far as I know.

Answer 3

I don't think you can avoid downloading them, zipping them locally, and uploading them back.

Compression utilities work with local resources only. Azure Storage itself has no concept/ability of being able to compress some files by itself

Answer 4

I'd be 99% sure that whatever zip library you're using will require local resources/local files in order to create a zip file.

Have a look at Azure Local Storage for Worker Roles.

http://vkreynin.wordpress.com/2010/01/10/learning-azure-local-storage-with-me/

You'll be able to specify an amount of local storage within your worker role in order to save content accessible in process.

e.g.

//Create a Local Storage section in your config.

<WebRole name="...">
 <LocalResources>
   <LocalStorage name="myLocalStorage" sizeInMB="50"/>
 </LocalResources>
</WebRole>

//Then save your files to local storage

CloudBlobContainer container = blobClient.GetContainerReference("myfiles");
CloudBlob blob = container.GetBlobReference("2b5f8ea6-3dc2-4b77-abfe-4da832e02556/AppList/isjirleq/mydocs1.doc");

LocalResource myStorage = RoleEnvironment.GetLocalResource("myLocalStorage");
string filePath = Path.Combine(myStorage.RootPath, "mydocs1.doc");
blob.DownloadToFile(filePath);

Once you've got all your files saved in LocalStorage, use your ZipLibrary to bundle up all your file paths together

Answer 5

I think you can use webjob or worker role to do it. When you receive a user's request, push this request into queue, then return the job id to user. Webjob or worker role fetch the request from queue, and download these files and zip them, then upload the zip file back to Storage blob. Front side code can use ajax roll polling with the job id to get the real download url when it is done.

Answer 6

I have done this Using JSZip from website to download a multiple files from Azure Blob Storage

var urls = [
"images/20170420_145140.jpg",
"images/20170503_142841.jpg",
"images/20170503_084035.jpg"];


download() {

    urls.forEach(function (url) {
        JSZipUtils.getBinaryContent(url, function (err, data) {
            if (err) {
                throw err; // or handle the error
            }
            try {
                zip.file(count + ".jpg", data, { binary: true });
                count++;
                if (count == urls.length) {
                    zip.generateAsync({ type: "blob" }).then(function (content) {
                        FileSaver.saveAs(content, zipFilename);
                    });
                }
            } catch (e) {
                console.log("errorrr...k", e)
            }
        });
    });
}

Answer 7

dont think azure provides any out of box functionality for zipping.If you need to avoid turnaround time maybe.using a background worker role be a good idea which will pick your files from certain queue.zip them upload them and store url for you somewhere like sql db.I have done similar things while communicating with db so you can be assured it will work very fast without user realising that its actually not happening at front end.