Write csv data directly to azure blob storage

Question

I have a large array with around 1 million data objects. I have found lots of samples for uploading files to azure blob storage. I guess you could do it with a memory stream, but I haven't found samples for doing it from objects. I am not sure with that size of data whether you should write line by line of what options I have. All input are welcome, would be perfect with some samples. The goal is to write the data objects to a csv file in Azure Blob Storage.

Answer 1

I assume you are writing the code in C# with the latest version of Azure Storage SDK for .NET (9.3.3).

Here is my code for realizing your needs to write a large array of data objects directly to Azure Blob Storage.

using System;
using System.Collections;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;

namespace WriteCSVDataToBlob
{

    class Record
    {
        string[] cols;

        public Record(string[] cols)
        {
            this.cols= cols;
        }

        override public string ToString()
        {
            return String.Join(',', cols);
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var arr = new ArrayList();
            arr.Add(new Record(new string[]{ "A", "B","one" }));
            arr.Add(new Record(new string[] { "C", "D","two"}));
            string storageConnectionString = "<your storage connection string>";
            var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
            var client = storageAccount.CreateCloudBlobClient();
            var container = client.GetContainerReference("test");
            var blob = container.GetBlockBlobReference("data.txt");
            using (CloudBlobStream x = blob.OpenWriteAsync().Result)
            {
                foreach(var rec in arr)
                {
                    x.Write(System.Text.Encoding.Default.GetBytes(rec.ToString()+"\n"));
                }
                x.Flush();
                x.Close();
            }
        }
    }
}

Answer 2

@Peter Pan's solution works for Microsoft.Azure.Storage.Blob v11.1.0. In newer version of azure blob storage you can use something like this:

                using (var writer = new StreamWriter(new MemoryStream()))
                using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
                {
                    csv.Configuration.Delimiter = userInputModel.Separator;
                    csv.WriteRecords(arr);
                    writer.Flush();

                    writer.BaseStream.Seek(0, SeekOrigin.Begin);

                    client.AppendBlock(writer.BaseStream);
                }