REST request on an Azure storage account using an access key results in HTTP 403 Forbidden

Question

I have an Azure storage account with no public blob access. I can access blob, table and query via the (.NET) APIs using one of the storage account access keys. For REST I tried the Microsoft demo application on https://docs.microsoft.com/en-us/azure/storage/common/storage-rest-api-auth, of course with my storage account name and one of the storage account access keys. This demo application just lists the blob containers. It results in a HTTP 403 (Forbidden) when trying to connect.

I cannot find a reason. Is the storage account access key the right key to use (I cannot create shared access signatiures for some reason to try them)? Ideas are appreciated.

Here is the complete code (please note that I replaced the storage account name and access key by "xxx"):

using System;
using System.Globalization;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Xml.Linq;

internal static class Program
{
    static string StorageAccountName = "xxx";
    static string StorageAccountKey = "xxx";
    
    private static void Main()
    {
        // List the containers in a storage account.
        ListContainersAsyncREST(StorageAccountName, StorageAccountKey, CancellationToken.None).GetAwaiter().GetResult();

        Console.WriteLine("Press any key to continue.");
        Console.ReadLine();
    }

    /// <summary>
    /// This is the method to call the REST API to retrieve a list of
    /// containers in the specific storage account.
    /// This will call CreateRESTRequest to create the request, 
    /// then check the returned status code. If it's OK (200), it will 
    /// parse the response and show the list of containers found.
    /// </summary>
    private static async Task ListContainersAsyncREST(string storageAccountName, string storageAccountKey, CancellationToken cancellationToken)
    {

        // Construct the URI. This will look like this:
        //   https://myaccount.blob.core.windows.net/resource
        String uri = string.Format("http://{0}.blob.core.windows.net?comp=list", storageAccountName);

        // Set this to whatever payload you desire. Ours is null because 
        //   we're not passing anything in.
        Byte[] requestPayload = null;

        //Instantiate the request message with a null payload.
        using (var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri)
        { Content = (requestPayload == null) ? null : new ByteArrayContent(requestPayload) })
        {

            // Add the request headers for x-ms-date and x-ms-version.
            DateTime now = DateTime.UtcNow;
            httpRequestMessage.Headers.Add("x-ms-date", now.ToString("R", CultureInfo.InvariantCulture));
            httpRequestMessage.Headers.Add("x-ms-version", "2017-04-17");
            // If you need any additional headers, add them here before creating
            //   the authorization header. 

            // Add the authorization header.
            httpRequestMessage.Headers.Authorization = AzureStorageAuthenticationHelper.GetAuthorizationHeader(
               storageAccountName, storageAccountKey, now, httpRequestMessage);

            // Send the request.
            using (HttpResponseMessage httpResponseMessage = await new HttpClient().SendAsync(httpRequestMessage, cancellationToken))
            {
                // If successful (status code = 200), 
                //   parse the XML response for the container names.
                if (httpResponseMessage.StatusCode == HttpStatusCode.OK)
                {
                    String xmlString = await httpResponseMessage.Content.ReadAsStringAsync();
                    XElement x = XElement.Parse(xmlString);
                    foreach (XElement container in x.Element("Containers").Elements("Container"))
                    {
                        Console.WriteLine("Container name = {0}", container.Element("Name").Value);
                    }
                }
            }
        }
    }
}

Answer 1

I think your storage account is set to only allow HTTPS which means that you will need to change the uri from HTTP to HTTPS.

Change this:

String uri = string.Format("http://{0}.blob.core.windows.net?comp=list", storageAccountName);

To this:

String uri = string.Format("https://{0}.blob.core.windows.net?comp=list", storageAccountName);

Doing this worked for me using your code, I only put in the storageaccount name and Access Key by copy paste from the portal.

Answer 2

Based on the Microsoft sample code, comments on my question, and the documentation on https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key, I developed a solution that also works for Table requests in the meantime. For others struggling with REST storage requests, here it is. Note that the authorization headers are for SharedKey authorization for the 2009-09-19 version and later of the Blob, Queue and Table services, and the 2014-02-14 version and later of the File services.

/// <summary>
/// Creates the authorization headers needed for Azure Storage REST calls.
/// </summary>
/// <remarks>
/// This class is bases on the Microsoft sample code on 
/// https://github.com/Azure-Samples/storage-dotnet-rest-api-with-auth
/// </remarks>
internal static class AzureStorageAuthenticationHelper
{
    /// <summary>
    /// Creates a SharedKey authorization header for blob, query and file requests according to 
    /// https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#blob-queue-and-file-services-shared-key-authorization.
    /// </summary>
    /// <param name="storageAccountName">The name of the storage account to use.</param>
    /// <param name="storageAccountKey">The access key for the storage account to be used.</param>
    /// <param name="now">Date/Time for now. Note that a request must not be older than 15 minutes. 
    /// Otherwise a HTTP 403 (Forbidden) results.</param>
    /// <param name="httpRequestMessage">The HttpWebRequest that needs an authorization header.</param>
    /// <param name="ifMatch">Provide an eTag, and a blob is only modified, if the current eTag matches. 
    /// This ensures that changes of others are not overwritten (provided, they add a eTag too).</param>
    /// <param name="md5">If not null the passed md5 ic checked if it matches the blob's md5. If the md5 does
    /// not match, the query will not return a value.</param>
    internal static AuthenticationHeaderValue GetAuthorizationHeaderForBlobAndQueueAndFile(string storageAccountName, string storageAccountKey,
       DateTime now, HttpRequestMessage httpRequestMessage, string ifMatch = "", string md5 = "")
    {
        // This is the raw representation of the message signature
        HttpMethod method = httpRequestMessage.Method;

        var headerContentLength = method == HttpMethod.Get || method == HttpMethod.Head
            ? String.Empty
            : httpRequestMessage.Content.Headers.ContentLength.ToString();
        var cannonicalHeaders = GetCanonicalizedHeaders(httpRequestMessage);
        var cannnonicalResource = GetCanonicalizedResourceForBlobAndQueue(httpRequestMessage.RequestUri, storageAccountName);

        // Create a signature according to https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#blob-queue-and-file-services-shared-key-authorization
        // for the 2009-09-19 version 
        var signatureStringToSign =
            method + "\n" +                 // VERB
            "\n" +                          // Content-Encoding 
            "\n" +                          // Content-Language
            headerContentLength + "\n" +    // Content-Length
            md5 + "\n" +                    // Content-MD5 
            "\n" +                          // Content-Type
            "\n" +                          // Date
            "\n" +                          // If-Modified-Since
            ifMatch + "\n" +                // If-Match
            "\n" +                          // If-None-Match
            "\n" +                          // If-Unmodified-Since
            "\n" +                          // Range
            cannonicalHeaders + cannnonicalResource;
        var storageAccountKeyMessageAuthenticationCode = new HMACSHA256(Convert.FromBase64String(storageAccountKey));
        var signature = Convert.ToBase64String(storageAccountKeyMessageAuthenticationCode.ComputeHash(
            Encoding.UTF8.GetBytes(signatureStringToSign)));

        // Create the actual header that will be added to the list of request headers
        var authenticationHeaderValue = new AuthenticationHeaderValue("SharedKey", storageAccountName + ":" + signature);

        return authenticationHeaderValue;
    }

    /// <summary>
    /// Creates a SharedKey authorization header for table requests according to 
    /// https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#table-service-shared-key-authorization
    /// </summary>
    /// <param name="storageAccountName">The name of the storage account to use.</param>
    /// <param name="storageAccountKey">The access key for the storage account to be used.</param>
    /// <param name="now">Date/Time for now. Note that a request must not be older than 15 minutes. 
    /// Otherwise a HTTP 403 (Forbidden) results.</param>
    /// <param name="httpRequestMessage">The HttpWebRequest that needs an authorization header.</param>
    /// <param name="ifMatch">Provide an eTag, and a blob is only modified, if the current eTag matches. 
    /// This ensures that changes of others are not overwritten (provided, they add a eTag too).</param>
    /// <param name="md5">If not null the passed md5 ic checked if it matches the blob's md5. If the md5 does
    /// not match, the query will not return a value.</param>
    internal static AuthenticationHeaderValue GetAuthorizationHeaderForTable(string storageAccountName, string storageAccountKey,
       DateTime now, HttpRequestMessage httpRequestMessage, string md5 = "")
    {
        // This is the raw representation of the message signature
        HttpMethod method = httpRequestMessage.Method;

        var headerContentLength = method == HttpMethod.Get || method == HttpMethod.Head
            ? String.Empty
            : httpRequestMessage.Content.Headers.ContentLength.ToString();
        var date = now.ToString("R", CultureInfo.InvariantCulture);
        var cannnonicalResource = GetCanonicalizedResourceForTable(httpRequestMessage.RequestUri, storageAccountName);

        // Create a signature according to https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#table-service-shared-key-authorization
        // for the 2009-09-19 version. Note that the date must be specified here and be the same as in the x-ms-date header.        
        var signatureStringToSign =
            method + "\n" +     // VERB
            md5 + "\n" +        // Content-MD5
            "\n" +              // Content-Type 
            date + "\n" +       // Date
            cannnonicalResource;
        var storageAccountKeyMessageAuthenticationCode = new HMACSHA256(Convert.FromBase64String(storageAccountKey));
        var signature = Convert.ToBase64String(storageAccountKeyMessageAuthenticationCode.ComputeHash(
            Encoding.UTF8.GetBytes(signatureStringToSign)));

        // Create the actual header that will be added to the list of request headers
        var authenticationHeaderValue = new AuthenticationHeaderValue("SharedKey", storageAccountName + ":" + signature);

        return authenticationHeaderValue;
    }

    /// <summary>
    /// Gets a canonical string for the x-ms headers of a HTTP request according to
    /// https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#constructing-the-canonicalized-headers-string.
    /// </summary>
    /// <remarks>
    /// A cannnical string is a string "in the right format".
    /// </remarks>
    /// </summary>
    /// <param name="httpRequestMessage">The request that will be made to the storage service</param>
    /// <returns>Error message in case of an exception</returns>
    private static string GetCanonicalizedHeaders(HttpRequestMessage httpRequestMessage)
    {
        // Get the x-ms headers with lowercase key and value
        var microsoftHeaders =
            from header in httpRequestMessage.Headers
            where header.Key.StartsWith("x-ms-", StringComparison.OrdinalIgnoreCase)
            orderby header.Key
            select new { Key = header.Key.ToLowerInvariant(), header.Value };


        // Create the string in the right format; this is what makes the headers "canonicalized",
        // meaning to put it in a standard format. See http://en.wikipedia.org/wiki/Canonicalization
        var resultBuilder = new StringBuilder();
        foreach (var microsoftHeader in microsoftHeaders)
        {
            var headerBuilder = new StringBuilder(microsoftHeader.Key);
            var separator = ':';

            // Get the value for each header, remove \r\n, and append the value separated by the current separator
            foreach (string headerValue in microsoftHeader.Value)
            {
                var trimmedValue = headerValue.TrimStart().Replace("\r\n", String.Empty);
                headerBuilder.Append(separator).Append(trimmedValue);

                // After the first value, set the separator to a comma
                separator = ',';
            }

            // Append the header
            resultBuilder.Append(headerBuilder.ToString()).Append("\n");
        }

        return resultBuilder.ToString();
    }

    /// <summary>
    /// Creates a canonical string representing the storage service resource targeted by the request according to
    /// https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#shared-key-format-for-2009-09-19-and-later
    /// for blob and queue services.
    /// </summary>
    /// <remarks>
    /// A cannnical string is a string "in the right format".
    /// </remarks>
    /// <param name="address">The URI of the storage service.</param>
    /// <param name="accountName">The storage account name.</param>
    /// <returns>String representing the canonicalized resource.</returns>
    private static string GetCanonicalizedResourceForBlobAndQueueX(Uri address, string storageAccountName)
    {
        // The absolute path is "/" because for we're getting a list of containers.
        var resultBuilder = new StringBuilder("/").Append(storageAccountName).Append(address.AbsolutePath);

        // Address.Query is the resource, such as "?comp=list".
        // This ends up with a NameValueCollection with 1 entry having key=comp, value=list.
        // It will have more entries if you have more query parameters.
        var queryValues = HttpUtility.ParseQueryString(address.Query);

        foreach (var item in queryValues.AllKeys.OrderBy(key => key))
        {
            resultBuilder.Append('\n').Append(item.ToLower()).Append(':').Append(queryValues[item]);
        }

        return resultBuilder.ToString();
    }

    /// <summary>
    /// Creates a canonical string representing the storage service resource targeted by the request according to
    /// https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#shared-key-format-for-2009-09-19-and-later
    /// for blob and queue services.
    /// </summary>
    /// <remarks>
    /// A cannnical string is a string "in the right format".
    /// </remarks>
    /// <param name="address">The URI of the storage service.</param>
    /// <param name="accountName">The storage account name.</param>
    /// <returns>String representing the canonicalized resource.</returns>
    private static string GetCanonicalizedResourceForBlobAndQueue(Uri address, string storageAccountName)
    {
        // 1. Beginning with an empty string (""), append a forward slash (/), followed by the name 
        // of the account that owns the resource being accessed.
        var resultBuilder = new StringBuilder($"/{storageAccountName}");

        // 2. Append the resource's encoded URI path, without any query parameters.
        resultBuilder.Append(address.AbsolutePath);

        // 3. Retrieve all query parameters on the resource URI, including the comp parameter if it exists.
        var queryValues = HttpUtility.ParseQueryString(address.Query);

        // 4. Convert all parameter names to lowercase.
        // 5. Sort the query parameters lexicographically by parameter name, in ascending order.
        // 6. URL-decode each query parameter name and value.
        // 7. Include a new-line character (\n) before each name-value pair.
        // 8. Append each query parameter name and value to the string in the following format, 
        //    making sure to include the colon (:) between the name and the value: parameter - name:parameter - value
        // 9. If a query parameter has more than one value, sort all values lexicographically, then include them in a comma-separated list:
        //    parameter - name:parameter - value - 1,parameter - value - 2,parameter - value - n
        foreach (var item in queryValues.AllKeys.OrderBy(key => key))
        {
            resultBuilder.Append('\n').Append(item.ToLower()).Append(':').Append(queryValues[item]);
        }

        return resultBuilder.ToString();
    }

    /// <summary>
    /// Creates a canonical string representing the storage service resource targeted by the request according to
    /// https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key#shared-key-lite-and-table-service-format-for-2009-09-19-and-later
    /// for table services.
    /// </summary>
    /// <remarks>
    /// A cannnical string is a string "in the right format".
    /// </remarks>
    /// <param name="address">The URI of the storage service.</param>
    /// <param name="accountName">The storage account name.</param>
    /// <returns>String representing the canonicalized resource.</returns>
    private static string GetCanonicalizedResourceForTable(Uri address, string storageAccountName)
    {
        // 1. Beginning with an empty string (""), append a forward slash (/), followed by the name 
        // of the account that owns the resource being accessed.
        var resultBuilder = new StringBuilder($"/{storageAccountName}");

        // 2a. Append the resource's encoded URI path. 
        resultBuilder.Append(address.AbsolutePath);

        // 2b. If the request URI addresses a component of the resource, append the appropriate query string. 
        // The query string should include the question mark and the comp parameter 
        // (for example, ?comp=metadata). No other parameters should be included on the query string.
        var queryValues = HttpUtility.ParseQueryString(address.Query);
        if (queryValues.AllKeys.Contains("comp"))
        {
            resultBuilder.Append($"?{queryValues["comp"]}");
        }

        return resultBuilder.ToString();
    }
}