Error 403 on Creating Azure Storage Blob Container (Storage Firewall Enabled

Question

Update:

After the comment from Maurad I found a log that shows me that the "CreateContainer" operations seems to be running under a different IP ???????? Any idea why this would happen?

Original Post:

I have an azure storage account with the Firewall enabled and I am trying to create a blob container in the account using a VSTS build pipeline.

The pipeline is ran by a Microsoft hosted agent, and because of that it is part of my process to add the IP of the machine to the firewall before creating the container, and removing it just after.

The problem is that I am getting an error 403 when I ran the create blob command.

I've tried adding a wait time after adding the ip to the firewall but, even if I wait for 5 minutes, I still get the 403.

This is the script that I am running:

$MyIP = (Invoke-WebRequest 'https://ifconfig.me/ip' -Method Get).Content

Try {
    Add-AzStorageAccountNetworkRule -ResourceGroupName $ResourceGroupName `
        -Name $StorageAccountName `
        -IPAddressOrRange $MyIP

    $ctx = (Get-AzStorageAccount -ResourceGroupName $ResourceGroupName `
            -Name $StorageAccountName).Context

    New-AzStorageContainer -Name $ContainerName `
        -Context $ctx
}
Finally {
    Remove-AzStorageAccountNetworkRule -ResourceGroupName $ResourceGroupName `
        -Name $StorageAccountName `
        -IPAddressOrRange $MyIP
}

and this is the result I get (you can see the ip of the agent is properly added)

Action IPAddressOrRange
------ ----------------
 Allow 127.0.0.1
 Allow 104.40.203.123 # This is the build agent IP

New-AzStorageContainer: C:\Users\Desktop\Test.ps1:15
Line |
  15 |      New-AzStorageContainer -Name $ContainerName `
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | This request is not authorized to perform this operation. HTTP Status Code: 403 - HTTP Error Message:
     | This request is not authorized to perform this operation.
ErrorCode: AuthorizationFailure
     | ErrorMessage: This request is not authorized to perform this operation.
     | RequestId:9112d81a-e01e-002a-7935-331d33000000
Time:2020-05-26T08:11:04.9195569Z

Action IPAddressOrRange
------ ----------------
 Allow 127.0.0.1

Remark: Disabling the firewall works, but I cannot do that in prod

If you have configured storage analytics on your storage account, you can check the analytics logs to see from which IP address the request for creating the blob container is coming. — Gaurav Mantri
I didn't, but I can do that. I'll try and update with my findings. Thanks — Lucas Araujo
I don't see any information about origin IP addresses on my analytics. Only usage metrics. — Lucas Araujo
Do you see a $logs container in your storage account (you'll need to use Microsoft Storage Explorer)? That's where you will see the analytics logs. Please see this for more information: docs.microsoft.com/en-us/rest/api/storageservices/…. — Gaurav Mantri
Found! Thanks! You seem to be right 🤔, for some really strange reason the "CreateContainer" operation has a different IP of ALL the other operations. Any idea why that would be? — Lucas Araujo

Lucas Araujo Lucas Araujo · Accepted Answer · 2020-06-24T09:30:21

I have an answer from Microsoft on this subject.

What is happening is that when we create an Azure DevOps organization, we need to inform a region for it, but these regions are a subset of the Azure regions. Whenever I ran a pipeline using a Microsoft Hosted Agent, that agent is spun up on some Azure region.

If the agent is located in the same region as the storage account, then the communication will happen using private IPs, which are not supported on the Storage Account whitelisting, thereby you have the error.

So that is it, there is no workaround to fix this using Microsoft Hosted Agents.

Alternative

As suggested by Microsoft, you will have to use a Self-hosted agent to ensure that this problem won't happen. No other known alternative.

Long Term Solution

There is a feature request to include Azure DevOps as one of the possible selections for "Trusted Services" on Storage account. This will fix the problem in a proper way, however there is no timeline for this feature to be implemented.

Hopefully this will help someone else with the same problem.

🤷‍♀️

Error 403 on Creating Azure Storage Blob Container (Storage Firewall Enabled

1 Answers

Alternative

Long Term Solution