403 error in production from WindowsAzure.Storage

Question

I have a WebForms app that uses the WindowsAzure.Storage API v3. It works fine in development and in one production environment, but I'm rolling out a new instance and any code that calls out Azure Blob Storage gives me a 403 error.

I've been fiddling with this for awhile, and it fails on any call out to Blob Storage, so rather than show my code I'll show my stack trace:

[WebException: The remote server returned an error: (403) Forbidden.]
   System.Net.HttpWebRequest.GetResponse() +8525404
   Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync(RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) +1541

[StorageException: The remote server returned an error: (403) Forbidden.]
   Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync(RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) +2996
   Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExists(BlobContainerPublicAccessType accessType, BlobRequestOptions requestOptions, OperationContext operationContext) +177
   ObsidianData.Azure.Storage.GetContainer(CloudBlobClient client, Containers targetContainer) in D:\Dev\nSource\Obsidian\Source\ObsidianData\Azure\Storage.vb:84
   ObsidianWeb.Leads.HandleListenLink(String fileName, HyperLink link) in D:\Dev\nSource\Obsidian\Source\ObsidianWeb\Bdc\Leads.aspx.vb:188
   ObsidianWeb.Leads.LoadEntity_ContactDetails(BoLead lead) in D:\Dev\nSource\Obsidian\Source\ObsidianWeb\Bdc\Leads.aspx.vb:147
   ObsidianWeb.Leads.LoadEntity(BoLead Lead) in D:\Dev\nSource\Obsidian\Source\ObsidianWeb\Bdc\Leads.aspx.vb:62
   EntityPages.EntityPage`1.LoadEntity() +91
   EntityPages.EntityPage`1.Page_LoadComplete(Object sender, EventArgs e) +151
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +4018

Here's what I've tried...

The AzureStorageConnectionString that fails in this environment definitely works in production
Other connection strings (from the other production environment, which works) also get a 403 here
There seemed to be an issue with timestamps in some old versions of the REST api (which I am not directly using...) so I made certain the times are correct, even tried switching the server to UTC time.
Tried toggling the connection string between http/https.
Upgraded to the latest version of the API (v3.1)
Tried fiddling with the code to ensure that every call out to Azure Storage gets 403. It does.
In desperation, Installed Azure Powershell on the server just to verify that some type of communication with Azure is working. And that worked fine.
Browsed to the azure management portal as well and that works fine.

Any ideas? This should just be using port 80 or 443, right? So there should be no way this is some kind of network issue. Let me know if that's wrong.

The working production machine is an Azure VM (Server 2008 R2 with IIS 7.5) There are also some differences with the server:
This new machine is physical hardware (Server 2012 and IIS 8)
This IS using a different storage account inside my azure subscription, however I've tried a total of 3 connection strings and none of them work here.

UPDATE: someone asked to see the code. Okay, I wrote a class called Azure.Storage, which just abstracts my cloud storage code. We are failing on a call to Storage.Exists, so here's the part of that class that feels relevant:

    Public Shared Function Exists(container As Containers, blobName As String) As Boolean
        Dim Dir As CloudBlobContainer = GetContainer(container)
        Dim Blob As CloudBlockBlob = Dir.GetBlockBlobReference(blobName.ToLower())

        Return Blob.Exists()
    End Function

    Private Shared Function GetContainer(client As CloudBlobClient, targetContainer As Containers)
        Dim Container As CloudBlobContainer = client.GetContainerReference(targetContainer.ToString.ToLower())
        Container.CreateIfNotExists()
        Container.SetPermissions(New BlobContainerPermissions() With {.PublicAccess = BlobContainerPublicAccessType.Blob})

        Return Container
    End Function

    Private Shared Function GetCloudBlobClient() As CloudBlobClient
        Dim Account As CloudStorageAccount = CloudStorageAccount.Parse(Settings.Cloud.AzureStorageConnectionString())

        Return Account.CreateCloudBlobClient()
    End Function

...Containers is just an enum of container names (there are several):

 Public Enum Containers
     CallerWavs
     CampaignImports
     Delve
     Exports
     CampaignImages
     Logos
     ReportLogos
     WebLinkImages
 End Enum

...Yes, they have upper-case characters, which causes problems. Everything is forced to lowercase before it goes out.

Also I did verify that the correct AzureConnectionString is coming out of my settings class. Again, I tried a few that work elsewhere. And this one works elsewhere also!

Please check the clock on the servers in question. You can also get 403 error if the time on the server is not in sync (+/- 15 minutes deviation is allowed) with the time on storage servers. — Gaurav Mantri
@GauravMantri Checked that, that's in the list I included in the question, but thanks. ;) — Brian MacKay
Another thing you could try is take the InnerException which should be a WebException, read its response stream to see more details about the error. — Gaurav Mantri
@GauravMantri Hrmmm, I'll try that. Do you think that would provide more than what I got in the stack trace? — Brian MacKay

Gaurav Mantri Gaurav Mantri · Accepted Answer · 2014-04-03T06:10:14

Please check the clock on the servers in question. Apart from the incorrect account key, you can also get 403 error if the time on the server is not in sync with the time on storage servers (Give or take +/- 15 minutes deviation is allowed).

403 error in production from WindowsAzure.Storage

3 Answers