Azure Data Lake : Facing issue while moving data from Blob to ADLS

Question

I am creating an Azure Function in C# which does following things:

extracts the zipped file from a blob,
unzips it and copies it to Azure Data Lake Store.

I was able to unzip the file and upload it into another blob using the UploadFromStreamAsync(stream) function.

However, I am facing issue while doing the same For ADLS

I referred to the below link Upload to ADLS from file stream and tried to first create the file using adlsFileSystemClient.FileSystem.Create and then append the stream using adlsFileSystemClient.FileSystem.Append in data lake but it did not work. - The create method creates a zero byte file but the append does nothing and the azure function still completes successfully without any error. Also, tried with adlsFileSystemClient.FileSystem.AppendAsync and still the same problem.

Code:

 // Save blob(zip file) contents to a Memory Stream.
    using (var zipBlobFileStream = new MemoryStream())
    {
        await blockBlob.DownloadToStreamAsync(zipBlobFileStream);
        await zipBlobFileStream.FlushAsync(); 
        zipBlobFileStream.Position = 0;

        //use ZipArchive from System.IO.Compression to extract all the files from zip file
        using (var zip = new ZipArchive(zipBlobFileStream))
        {
        //Each entry here represents an individual file or a folder
        foreach (var entry in zip.Entries)
        {
            string destfilename = $"{destcontanierPath2}/"+entry.FullName;
            log.Info($"DestFilename: {destfilename}");
            //creating an empty file (blobkBlob) for the actual file with the same name of file
            var blob = extractcontainer.GetBlockBlobReference($"{destfilename}");
            using (var stream = entry.Open())
            {
                //check for file or folder and update the above blob reference with actual content from stream
                if (entry.Length > 0)
                {
                    await blob.UploadFromStreamAsync(stream);

                    //Creating a file and then append
                    adlsFileSystemClient.FileSystem.Create(_adlsAccountName, "/raw/Hello.txt",overwrite:true); 
                    // Appending the stream to Azure Data Lake 
                    using(var ms = new MemoryStream())
                    {
                        stream.CopyTo(ms);
                        ms.Position = 0; // rewind
                        log.Info($"**********MemoryStream: {ms}");
                        // do something with ms
                        await adlsFileSystemClient.FileSystem.AppendAsync(_adlsAccountName, "/raw/Hello.txt",ms,0);
                    }
                }
            }                           
        }
        }
    }

New Interim Solution:

    using (var zipBlobFileStream = new MemoryStream())
    {
        await blockBlob.DownloadToStreamAsync(zipBlobFileStream);
        using (var zip = new ZipArchive(zipBlobFileStream))
        {
            //Each entry here represents an individual file or a folder
            foreach (var entry in zip.Entries)
            {   
                    entry.ExtractToFile(directoryPath + entry.FullName, true);
                    //Upload the File to ADLS
                    var parameters = new UploadParameters(directoryPath + entry.FullName, "/raw/" + md5, _adlsAccountName, isOverwrite: true, maxSegmentLength: 268435456 * 2);
                    var frontend = new Microsoft.Azure.Management.DataLake.StoreUploader.DataLakeStoreFrontEndAdapter(_adlsAccountName, adlsFileSystemClient);
                    var uploader = new DataLakeStoreUploader(parameters, frontend);
                    uploader.Execute();
                    File.Delete(directoryPath + entry.FullName);

            }
        }
    }

Tom Sun - MSFT Tom Sun - MSFT · Accepted Answer · 2018-03-13T10:15:50

In your case, You could change your code as following, then it should work. You should remove the create file code out of the foreach clause.

//Creating a file and then append
adlsFileSystemClient.FileSystem.Create(_adlsAccountName, "/raw/Hello.txt",overwrite:true); 

 using (var zipBlobFileStream = new MemoryStream())
    {
        await blockBlob.DownloadToStreamAsync(zipBlobFileStream);
        await zipBlobFileStream.FlushAsync(); 
        zipBlobFileStream.Position = 0;

        //use ZipArchive from System.IO.Compression to extract all the files from zip file
        using (var zip = new ZipArchive(zipBlobFileStream))
        {
        //Each entry here represents an individual file or a folder
        foreach (var entry in zip.Entries)
        {
            string destfilename = $"{destcontanierPath2}/"+entry.FullName;
            log.Info($"DestFilename: {destfilename}");
            //creating an empty file (blobkBlob) for the actual file with the same name of file
            var blob = extractcontainer.GetBlockBlobReference($"{destfilename}");
            using (var stream = entry.Open())
            {
                //check for file or folder and update the above blob reference with actual content from stream
                if (entry.Length > 0)
                {
                    using (MemoryStream ms = new MemoryStream())
                            {
                                stream.CopyTo(ms);
                                ms.Position = 0;
                                blob.UploadFromStream(ms);
                                ms.Position = 0;                                   
                      adlsFileSystemClient.FileSystem.Append(adlsAccountName, "/raw/Hello.txt", ms);

                            }
                  }
                }
            }                           
        }
        }
    }

Azure Data Lake : Facing issue while moving data from Blob to ADLS

1 Answers