Performance when copying many small files from an Azure VM Temporary Disk to Azure Blobs

6
votes

I'm trying to figure out the best performing approach when writing thousands of small Blobs to Azure Storage. The application scenario is the following:

  • thousands of files are being created or overwritten by a constantly running windows service installed on a Windows Azure VM
  • Writing to the Temporary Storage available to the VM, the service can reach more than 9,000 file creations per second
  • file sizes range between 1 KB and 60 KB
  • on other VMs with same sw running, other files are being created with same rate and criteria
  • given the need to build and keep updated a central repository, another service running on each VM copies the newly created files from the Temporary Storage to Azure Blobs
  • other servers should then read the Azure Blobs in their more recent version

Please note that for many constraints that I'm not listing for shortness, it's not currently possible to modify the main service to directly create Blobs instead of files on Temporary file system. ...and from what I' currently seeing it would mean a slower rate of creation, not acceptable per original requirements.

This copy operation, that I'm testing in a tight loop on 10,000 files, seems to be limited at 200 blob creations per second. I've been able to reach this result after tweaking the sample code named "Windows Azure ImportExportBlob" found here: http://code.msdn.microsoft.com/windowsazure/Windows-Azure-ImportExportB-9d30ddd5 with the async suggestions found in this answer: Using Parallel.Foreach in a small azure instance

I obtained this apparent maximum of 200 blob creations per second on an extralarge VM with 8 cores and setting the "maxConcurrentThingsToProcess" Semaphore accordingly. The network utilization during the test is max 1% of the available 10Gb shown in task manager. This means roughly 100 Mb of the 800 Mb that should be available on that VM size.

I see that the total size copied during the elapsed time gives me around 10 MB/sec.

Is there some limitation on the Azure Storage traffic you can generate or should I use a different approach when writing so many and small files ?

1
azureazure-storage-blobs
Are the containers different for Azure Storage? Also do you have any worker queues set up that you can scale out to, so they can distribute the copy/serialize/writes? Also is your VM attached storage striped in a RAID? (there are hacks to make this faster BTW, but MSFT is going to probably release VMs similar to what AWS has for specific scenarios: high I/O in your case) – Bart Czernicki
my answers to your 3 questions below: 1) Are the containers different for Azure Storage? No, my test program is writing all 10,000 files to the same container. This reminds me that I did not state that all files are written in a folder structure that is reproduced inside the container. Do you mean that a consistent improvement could be obtained writing to different containers? Do you have any url to documentation about this aspect? – MarioCannistra
2) Also do you have any worker queues set up that you can scale out to, so they can distribute the copy/serialize/writes? My first trial has been a parallel.for loop on the 10,000 files and I got just 3 files/sec. So I applied the async pattern described in the mentioned answer link and thus I'm writing with a concurrency of 8. This way I don't see cpu usage above 8% and I reached 200 files / sec with just 1% of the netowrk used. Increasing concurrency on the same VM does not increase the throughput – MarioCannistra
Seems that I'm hitting a limit in the underlying blob service and/or some contentione happens when "too many" threads are writing blobs. Please, consider as well that each of the VM writing those files is already the implementation of a scale out approach but I'm finding a bottleneck in the distributed storage usage. – MarioCannistra
3) Also is your VM attached storage striped in a RAID? I'm not using a VM attached storage for the source files to be copied. I'm using the Temporary drive, local to the VM node, that is automatically attached when you create the VM. On the extralarge VM it is a 600 GB drive. I'm satisfied with the more then 9,000 file creations / sec achieved on that. The Temporary storage has been mentioned by Russinovich in the TechEd Europe 2012 session where he also stripes 16 drives of 1TB each. – MarioCannistra

1 Answers

2
votes

@breischl Thank you for the scalability targets. After reading that post, I started searching for more target figures possibly prepared by Microsoft and found 4 posts (too many for my "reputation" to be posted here, the other 3 are part 2, 3 and 4 of the same series):

http://blogs.microsoft.co.il/blogs/applisec/archive/2012/01/04/windows-azure-benchmarks-part-1-blobs-read-throughput.aspx

the first post contains an important hint: "You may have to increase the ServicePointManager.DefaultConnectionLimit for multiple threads to establish more than 2 concurrent connections with the storage."

I've set this to 300 , rerun the test and seen an important increase in the MB/s. As I previously wrote, I was thinking to be hitting a limit in the underlying blob service when "too many" threads are writing blobs. This is the confirmation of my worries. Thus, I removed all the changes made to the code to work with a semaphore and replaced it again with a parallel.for to start as many blob upload operations as possible. The result has been awesome: 61 MB/s writing blobs and 65 MB/s reading.

The scalability target is 60 MB/s and I'm finally happy with the result.
Thank you all again for your answers.