Getting 403 when Azure REST API for blob storage with query strings having prefix or marker

Question

I am trying to use the Azure REST API for blob storage, and am using the AzureStorageAuthenticationHelper from https://github.com/Azure-Samples/storage-dotnet-rest-api-with-auth to make the authorization header.

If I do this:

private static void DoItViaRest(string containerName, ILogger log)
{
    try
    {
        string uri = string.Format("https://{0}.blob.core.windows.net/{1}?restype=container&comp=list", STORAGE_ACCOUNT_NAME, containerName);
        byte[] requestPayload = null;
        string xmlString = CallStorageRESTAPI(uri, requestPayload, log).Result;
    }
    catch (Exception e)
    {
        // handle exception
    }

    private static async Task<string> CallStorageRESTAPI(string uri, byte[] requestPayload, ILogger log)
    {
        string response = string.Empty;

        using (var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri) { Content = (requestPayload == null) ? null : new ByteArrayContent(requestPayload) })
        {
            DateTime now = DateTime.UtcNow;
            httpRequestMessage.Headers.Add("x-ms-date", now.ToString("R"));
            httpRequestMessage.Headers.Add("x-ms-version", "2017-07-29");
            httpRequestMessage.Headers.Authorization = AzureStorageAuthenticationHelper.GetAuthorizationHeader(STORAGE_ACCOUNT_NAME, STORAGE_ACCOUNT_KEY, now, httpRequestMessage);
            using (HttpResponseMessage httpResponseMessage = await new HttpClient().SendAsync(httpRequestMessage))
            {
                if (httpResponseMessage.StatusCode == HttpStatusCode.OK)
                {
                    response = await httpResponseMessage.Content.ReadAsStringAsync();
                }
                else
                {
                    log.LogInformation(string.Format("REST returned {0}", httpResponseMessage.StatusCode.ToString()));
                }
            }
        }
        return response;
    }

then the authorization created by AzureStorageAuthenticationHelper.GetAuthorizationHeader(STORAGE_ACCOUNT_NAME, STORAGE_ACCOUNT_KEY, now, httpRequestMessage) works fine.

However if I add prefix or marker to the URI's query string, then I end up with a 403. Examples with prefix: https://{0}.blob.core.windows.net/{1}?restype=container&comp=list&prefix={2} https://{0}.blob.core.windows.net/{1}?restype=container&prefix={2}&comp=list Examples with marker: https://{0}.blob.core.windows.net/{1}?restype=container&comp=list&marker={2} https://{0}.blob.core.windows.net/{1}?restype=container&marker={2}&comp=list

The marker value above is the NextMarker returned from the original call.

I am a loss for what's wrong or how to fix it. Suggestions?

Gaurav Mantri Gaurav Mantri · Accepted Answer · 2020-02-13T16:47:42

I believe you have discovered an issue with the code in the Github repo.

Essentially the problem is coming because of the following lines of code:

    foreach (var item in values.AllKeys.OrderBy(k => k))
    {
        sb.Append('\n').Append(item).Append(':').Append(values[item]);
    }

    return sb.ToString().ToLower();

Essentially what this code is doing is taking all the query string parameters (name and value) and appending them in name:value format and then finally converting the entire string to lowercase (and that's the problem).

According to documentation, one should only be converting the key name to lowercase and not the value portion.

To fix this issue, please use the following code:

    /// <summary>
    /// This part of the signature string represents the storage account 
    ///   targeted by the request. Will also include any additional query parameters/values.
    /// For ListContainers, this will return something like this:
    ///   /storageaccountname/\ncomp:list
    /// </summary>
    /// <param name="address">The URI of the storage service.</param>
    /// <param name="accountName">The storage account name.</param>
    /// <returns>String representing the canonicalized resource.</returns>
    private static string GetCanonicalizedResource(Uri address, string storageAccountName)
    {
        // The absolute path is "/" because for we're getting a list of containers.
        StringBuilder sb = new StringBuilder("/").Append(storageAccountName).Append(address.AbsolutePath);

        // Address.Query is the resource, such as "?comp=list".
        // This ends up with a NameValueCollection with 1 entry having key=comp, value=list.
        // It will have more entries if you have more query parameters.
        NameValueCollection values = HttpUtility.ParseQueryString(address.Query);

        foreach (var item in values.AllKeys.OrderBy(k => k))
        {
            sb.Append('\n').Append(item.ToLower()).Append(':').Append(values[item]);
        }

        return sb.ToString();//We should not be converting entire thing to lower case.
    }

However, please note that this is not still 100% correct as it doesn't take into consideration point #9 mentioned in the screenshot above but for your purpose it should work.

Getting 403 when Azure REST API for blob storage with query strings having prefix or marker

1 Answers