KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

Question

I'm trying to set up my web app, hosted in Azure to read settings from Azure KeyVault.

I've been following this guide: https://anthonychu.ca/post/secrets-aspnet-core-key-vault-msi/

The example shows how to access app settings from KeyVault with the configuration:

public static IWebHost BuildWebHost(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
     .ConfigureAppConfiguration((ctx, builder) =>
     {
         var keyVaultEndpoint = Environment.GetEnvironmentVariable("KEYVAULT_ENDPOINT");
         if (!string.IsNullOrEmpty(keyVaultEndpoint))
         {
             var azureServiceTokenProvider = new AzureServiceTokenProvider();
             var keyVaultClient = new KeyVaultClient(
                 new KeyVaultClient.AuthenticationCallback(
                     azureServiceTokenProvider.KeyVaultTokenCallback));
             builder.AddAzureKeyVault(
                 keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
         }
     })
    .UseApplicationInsights()
    .UseStartup<Startup>()
    .Build();

I've added the KEYVAULT_ENDPOINT environment variable to the application settings. I've enabled MSI on the app service, and I've authorized my Azure User, and my application, from the Key Vault Access Policies:

With Get and List Operations:

And I've added the secret to key vault. Running locally, I can access the secret.

But my ASP .NET Core site fails on startup with this in the stdout logs:

Unhandled Exception: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'
   at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Blog.Program.BuildWebHost(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 22
   at [csproj name].Program.Main(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 16

I've checked that MSI_ENDPOINT and MSI_SECRET environment variables exist by calling SET from the debug console.

I can also see the KEYVAULT_ENDPOINT variable.

Any suggestions what could be going wrong or on what to try next? Since it works locally it must be an authentication issue, but I believe it's authenticating locally with my azure User that I've authorized in key vault, rather than as the Azure App Service.

Answer 1

It is dodgy, but the problem with your setup is that you enabled "Authorized application" when you created the Access policy for your KeyValut.

You can see it because you have "Application+Application" on your screenshot. I imagine, you created the policy with your webapp being both principal and authrorized application. This kinda creates a loop.

To make this setup work just delete your existing policy and create a new one, where you select only the Principal:

Answer 2

only select principal and add web app,don't select authorized application

Answer 3

For me to resolve the issue, I had to add the IP addresses of the web app to the firewall as not all of them were getting added when you select "Allow trusted Microsoft services..."

Thanks to: https://azidentity.azurewebsites.net/post/2019/01/03/key-vault-firewall-access-by-azure-app-services

Answer 4

Check to ensure that the Service Principal you are using has been added and has the appropriate permissions for the operations you want to perform (Keys/Secrets etc):

Check to see if the KeyVault has restricted network access, toggle to test if this is the issue or not:

Answer 5

I have seen issues like this before, with development and production systems not behaving the same way when dealing with environment variables.

My first area to look at in these scenarios is to check if you have properly prefixed your environment variables with ASPNETCORE_ in your production system. This prefix is the default prefix for ASP.NET Core web hosts.

| Key                                |  Value                     |
|====================================|============================|
| ASPNETCORE_KEYVAULT_ENDPOINT       |  xxxxxxxxxxxxxxxx          |
| ASPNETCORE_MSI_ENDPOINT**          |  xxxxxxxxxxxxxxxx          |
| ASPNETCORE_MSI_SECRET**            |  xxxxxxxxxxxxxxxx          |

** Note that in your code I don't ever see you using MSI_ENDPOINT or MSI_SECRET. I only see you using KEYVAULT_ENDPOINT

If you want to explicitly set the prefix to something else, the prefix you want to use can be passed as an argument when initializing the web host via the configuration.

.AddEnvironmentVariables("ASPNETCORE_"); // choose your own prefix here

Answer 6

I want to add another case here with the same symptoms/error messages. Azure keyvault returned Forbidden when one of the secrets was disabled.

I had all the required permissions - I was accessing keyvault with Get and List granted for secrets (I was using only secrets) for the Managed Identity of my App Service.

It was confusing to get the "Access Forbidden" error, and it took some experimentation to figure out the issue. Hope this helps someone who have faced the same issue.

Answer 7

In the AZ KV, access policies... by selecting only the principal should work.

Answer 8

Another possibility is that the service principal needs API permissions for Key Vault. In Azure portal, go to Service Principal, then choose API permissions on the left hand menu. Then click Add permissions. Here you can choose Key Vault and save. This will allow the code to retrieve the key vault secrets.

Answer 9

If you added the access policy while you were logged into Azure through Visual Studio, you'll have to log out and log back in again to refresh your permissions.