Azure blob service, randomly 404-not found errors

Question

We have a web app which processes external callbacks. To isolate our app from the external service, we use an azure blob to store the callback data (.json) and put a message to Azure Service bus for a processing service to pick up later. The following code is used to write the data to blob storage:

        var storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("MyStorage"));
        var blobClient = storageAccount.CreateCloudBlobClient();
        var container = blobClient.GetContainerReference("containername");
        var dataFileName = Guid.NewGuid().ToString();
        var blockBlob = container.GetBlockBlobReference(dataFileName);
        blockBlob.UploadText(data);
        blockBlob.Properties.ContentType = "application/json";
        blockBlob.SetProperties();

        var connectionString = CloudConfigurationManager.GetSetting("serviceBusCS");
        var queueName = "MyQueue";
        var client = QueueClient.CreateFromConnectionString(connectionString, queueName);

        var payload = new MyCustomMessage {
            Id = dataFileName
        };
        var message = new BrokeredMessage(payload);
        client.Send(message);

On the processing side, we have a web job reading the messages one at a time and retrieving the data as follow:

            var storageAccount = CloudStorageAccount.Parse("MyCS");
            var blobClient = storageAccount.CreateCloudBlobClient();
            var container = blobClient.GetContainerReference("containername");
            var blockBlob = container.GetBlockBlobReference(message.Id);
            if (blockBlob == null) {
                return;
            }
            if (!blockBlob.Exists()) {
                return; ==> FAILS HERE
            } 
           // Process the message here...
           // Once the processing is done, delete the blob

This design works well most of the time but we get a 404-NotFound from now and then (marked with FAILED HERE above).

QUESTION

The only way this code can fail is by having two messages with the same file name i.e: the same GUID which is nearly impossible or am I missing something? Any idea why the blob data can't be found?

EDIT 1 Searching for the missing blob from the Azure portal shows that the blob is actually not there. Shouldn't blockBlob.UploadText(data); throw if it fails to write data to the blob container?

EDIT 2

Thanks to Jason for indicating where to look. We crawled our logs and found that the blob is getting written successfully. The web job kicks off and processes the message but one minute later, exactly one minute later we see the web job kicking off trying to process the very same message again and it won't find the blob which is expected as the first run cleared the blob and deleted it.

Jason Hogg - MSFT Jason Hogg - MSFT · Accepted Answer · 2016-10-05T21:01:01

If the client application receives an HTTP 404 (Not found) message from the server, this implies that the object the client was attempting to use (such as an entity, table, blob, container, or queue) does not exist in the storage service. There are a number of possible reasons for this, such as: •The client or another process previously deleted the object •A Shared Access Signature (SAS) authorization issue •Client-side JavaScript code does not have permission to access the object •Network failure

See Storage Monitoring, Diagnosing and Troubleshooting Guide for more information.

Azure blob service, randomly 404-not found errors

1 Answers