Unable to use MSI with AKS on Azure

Question

I have a k8s cluster in Azure using the AKS preview.

I also have MSI enabled on the VMs. If I ssh to the VMs I can see the MSI service is working:

curl http://localhost:50342/oauth2/token --data "resource=https://vault.azure.net" -H Metadata:true

I get a response that has what you'd expect based on the documentation.

However, I cannot figure out how to get my aspnet core applications to use the AzureServiceTokenProvider as desired. Locally it works, once deployed inside a pod in kubernetes it cannot find the authorization required.

The error I get is:

Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/9d93c307-6856-4bab-8fa9-99690e0fabaf. Exception Message: Tried the following 3 methods to get an access token, but none of them worked. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/9d93c307-6856-4bab-8fa9-99690e0fabaf. Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/9d93c307-6856-4bab-8fa9-99690e0fabaf. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Environment variable LOCALAPPDATA not set. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/9d93c307-6856-4bab-8fa9-99690e0fabaf. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. /bin/bash: az: No such file or directory ```

Has anyone been able to read from KeyVault in a pod using similar code?

AzureServiceTokenProvider azureServiceTokenProvider = new 
AzureServiceTokenProvider();
KVC = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var baseUrl = config.GetValue<string>("Azure:VaultURL");
var secretLocation = config.GetValue<string>("Azure:SecretLocation");
location = $"{baseUrl}{secretLocation}";
KVC.GetSecretAsync(location).Result.Value;

Isn't localhost, the container itself when you're in a pod, and not the VM? — evilSnobu
Yes. I’ve set up a Daemon set with host networking to have access to the VM localhost. What I cannot figure out is how to get the MSI_SECRET and MSI_ENDPOINT set. My plan is to proxy to the Daemon set for the pods on the node. SSH to the node and I do not see those environment vars set. But I can hit the oauth service. — Garry Polley
Seems a bit excessive. Why not just store the Azure AD app registration's client_id and client_secret as secrets in the cluster, then pass them into whatever pod needs them as env vars. You do need to acquire the access token from login.microsoftonline.com in that case with client credentials flow. — evilSnobu
I’m wanting to avoid having those secrets at all. My understanding is the purpose of MSI is to remove the need for hard coded secrets. My last resort will be hard coding and storing those values. — Garry Polley
Try this - github.com/Azure/acs-engine/blob/master/docs/kubernetes/…. A little bit alpha but might be enough. — evilSnobu

droessmj droessmj · Accepted Answer · 2019-04-05T00:12:28

Do you have any familiarity with AAD Pod Identity? https://github.com/Azure/aad-pod-identity. We've been working with this in an enterprise environment and user assigned identities, as opposed to an MSI.

This approach doesn't solve your MSI problem, but it does offer an alternative which may solve your underlying issue.

Unable to use MSI with AKS on Azure

1 Answers