Where should my Azure cloud service store files?

2
votes

I have a Azure cloud service which needs to access (read) thousands of small binary files and perform some computations with those. The key requirement here is speed - it needs to be as fast as possible. I tried using Azure blob storage for keeping these file but access was too slow, so currently I am keeping these files in the Azure cloud service local storage itself. This is not ideal as we are not supposed to store state in the VM -- persistence is not guaranteed and it is not shareable across VM instances.

I read about Azure premium storage (SSD drives which can be attached to VMs). This looked ideal in theory but I cannot see a way to add this to a cloud service.

So, my questions are:

  1. Is there a way to add Azure premium storage SSD disks to a cloud service or it can only be added to a Azure VM?
  2. Given my need to quickly read thousands of small binary files from a cloud service, are there any other storage mechanism I can try?
2
azureazure-storageazure-cloud-services
Is it a requirement to use CS instead of VM? – Alex Belotserkovskiy
Change your code to fit into the cloud's programming model, don't try to make the cloud work as a local server. If you want to process events, you should be using Event Hub and/or Stream Analytics. Blob storage scales to a lot of files but trying to process many small files isn't optimal - you end up paying more for network round-trips than actually loading data. Batch multiple messages into bigger files. Big data solutions like Hadoop love big files but hate many small ones - you can't partition a small file – Panagiotis Kanavos

2 Answers

1
votes

adding disks can be done to azure computer VMs and not cloud services. In this case you can use Azure file storage service which offers storage similar to network share. https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-files/

0
votes

Depending on your requirements, you could use Redis: https://azure.microsoft.com/en-us/services/cache/

It's very fast, and data stored on a Redis server can be shared across multiple VM instances.

Redis is a key-value pair store so you could use your files' "paths" as the keys and their contents as the values.

To minimize costs, make sure to locate your Redis server in the same region / resource group as your service.