Mysterious disappearing Azure development storage assets

Question

We're building a site backed by Azure Storage. One worker role has a few files that it downloads from the blob when it's starting up. The files are never modified once they're in storage, we just pull them down and use them.

On occasion, when trying to download these files from development storage the Storage Emulator service returns 500 errors. We can list the files in the blob and get metadata, but not download the file itself. The only solution we've found is to delete the blob and reupload.

Has anyone else run into this?

Update: 1.7 SDK

Answer 1

Possible Solution (Work Around)

1. Exit Storage Emulator;
2. Open up administrative Sql Server Management Studio 2012;
3. Attach C:\Users\<username>\DevelopmentStorageDb201206.mdf file (where <username>is the name of the affected user);
4. If it doesn't allow to attach, copy mdf and log files to some other drive and then attach, otherwise you can access to LocalDB ((localdb)\v11.0);
5. Find stored procedure CommitBlockList;
6. Change SET UncommittedBlockIdLength = NULL to SET UncommittedBlockIdLength = 0;
7. Execute it;
8. Close Management Studio;
9. Copy these edited mdf, log files to original location;
10. Start Storage Emulator;

How I got there

I found that about every seven days ONLY BLOCK blobs got deleted.
Creating those blobs again for testing purposes was painful while in the middle of development/testing.
Tried to find Storage Emulator source code but couldn't find it.
Turned on logging at C:\Users\<username>\AppData\Local\DevelopmentStorage by adding following to DevelopmentStorage.201206.config

<LogPath>C:\Users\<username>\AppData\Local\DevelopmentStorage\Logs</LogPath>  
<LoggingEnabled>true</LoggingEnabled>  

After painful waiting found following in logs:

DefragmentBlobFiles BlobInfo Name 40f5e12f-65a5-4a3a-ae46-41c71c8514c0/file1.txt, ContainerName storage1, Directory c:\users\username\appdata\local\developmentstorage\ldb\blockblobroot\1\12735b4b-f9ed-481b-a091-78387facf05b, ROFile , RWFile c:\users\username\appdata\local\developmentstorage\ldb\blockblobroot\1\12735b4b-f9ed-481b-a091-78387facf05b\1, Size5

I don't think above defragmentation causing any problems.
Found another log:

BlockBlob: Load Interval failed. IsGC: True, Exception at System.Number.ParseDouble(String value, NumberStyles options, NumberFormatInfo numfmt) at Microsoft.WindowsAzure.DevelopmentStorage.Store.BlockBlobGarbageCollector.GetTimerIntervalOrDefault(Boolean isGC)

So for BlockBlobs Uncommitted blocks are garbage-collected by this BlockBlobGarbageCollector. Nowhere I could find how often this uncommitted blocks are garbage-collected. I don't think even this is causing the problem.

Another log:

BlockBlob: Checking Directory C:\Users\username\AppData\Local\DevelopmentStorage\LDB\BlockBlobRoot\1\0477877c-4cb3-4ddb-a035-14a5cf52d86f in the list of valid directories
BlockBlob: Deleting Directory C:\Users\username\AppData\Local\DevelopmentStorage\LDB\BlockBlobRoot\1\0477877c-4cb3-4ddb-a035-14a5cf52d86f

THIS ABOVE LOG SHOWS THE PROBLEM. The emulator must be determining valid blockblob directories.

Checked schema of database DevelopmentStorageDb201206. Found few columns like IsCommitted and UncommittedBlockIdLength. Found that ClearUncommittedBlocks is setting UncommittedBlockIdLength to null. Any Blobs which were not being deleted were having UncommittedBlockIdLength value 0. So checked stored procedure CommitBlockList and changed UncommittedBlockIdLength to 0 instead of null. I think emulator in previous version must be checking IsCommitted and UncommittedBlockIdLength both to determine valid blockblob directories, while in this version it might be checking only UncommittedBlockIdLength as null and deleting all those block blob files.

As I said, it takes about seven days to find out whether this solution permanently fixes it. I have 4 more days to go to validate it.

If this is a workaround that works,... Microsoft owes me 6 hours;)

Answer 2

I've ran into the same issue. I suspect there is something wrong with Azure storage emulator when it is run on top of LocalDB. Here's why:

• when you got 500 error, open Blob table using Management Studio or Server Explorer, and find value of DirectoryPath field for blob in question (it will be something like c:\users\username\appdata\local\developmentstorage\ldb\blockblobroot\1\305469d0-7b68-4b1e-a41a-a429c21b6b9d

• navigate to that path using Explorer and notice that this directory is empty

• now reupload your file and navigate to a new directory it was uploaded to, notice that there is your file

So, the question now is why do blob files disappear?

Answer 3

I don't have a solution, but I can add that this behavior also occurs when the the storage emulator's backing store is SQL Server 2012. We've seen this repeatedly. All is well for a while, then blobs disappear. Our experience is that all, or nearly all blobs vanish from the file system while the database references persist. No clue why this happens - there is no obvious precipitating event.

Answer 4

I’ve also seen Storage Emulator occasionally returns for valid requests. But usually retry the request will work fine. It is always recommended to retry a failed request unless the response indicates the request is invalid (in which case retry won’t work anyway). Even if you use cloud storage, occasionally it may still encounters issue such as temporary network unavailable. As a matter of fact, retry policy is recommended for all network related operations.

In addition, if you use the .NET storage client library, you can take advantage of the built-in retry policy: http://blogs.msdn.com/b/windowsazurestorage/archive/2011/02/03/overview-of-retry-policies-in-the-windows-azure-storage-client-library.aspx.

Best Regards,

Ming Xu.

Answer 5

Perhaps the simpler way to solve this problem is to upgrade to the Azure SDK 1.8 -- since I've upgraded from 1.7 (from 2012-11-23, about two weeks ago) my blobs no longer disappear.

You can take advantage of the new storage emulator even if decide to use the old libraries -- indeed the libraries are installed side by side, only the emulators are upgraded (i.e. the new versions replace the older ones) but they are still compatible with older versions. For example, I'm now using the new storage emulator but I'm still referencing the 1.7 libraries in my .NET projects.

Update 2013-08-30 18:00 UTC I have this problem again with Azure SDK 2.0 -- for several weeks now I write blobs and after some minutes I get HTTP 500 when I access them. I wonder if this behavior is triggered by some characteristics of my storage emulator (for example too many blobs and/or container) -- with this regard I want to add that I upgraded from Storage Emulator 1.8 by renaming the database files since the schema (compared with Visual Studio) seemed to be the same, but maybe I missed something important.

I'm now evaluating whether to upgrade to Azure SDK 2.1 and hope that this issue is fixed or switch to use real storage accounts for development.

Update 2013-09-02 18:00 UTC I've decided to use a real storage account for the following reasons:

• the ability to maintain stored data through Azure SDK updates -- often the storage emulator changes the schema or the location of the database without any note in the release note, forcing to do some manual migration;
• being able to use all the features of Azure Storage without fear of disparity between production and storage emulator (as Richard Astbury said in his comment);
• avoid performance and maintenance problems peculiar to the SQL Server implementation (to me this is an accidental complexity).