How can I find the service principal secret of my AKS cluster?

Question

Okay, so I messed up, I accidentally ran az ad sp reset-credentials against the Service Principal that our AKS cluster runs under. And now we are getting errors like:

Error creating load balancer (will retry): error getting LB for service test/admin-api: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/resourceGroups/MC_****/providers/Microsoft.Network/loadBalancers?api-version=2017-09-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.\r\nTrace ID:****\r\nCorrelation ID:**** \r\nTimestamp: 2018-08-23 12:01:33Z","error_codes":[70002,50012],"timestamp":"2018-08-23 12:01:33Z","trace_id":"****","correlation_id":"****"}

and

Failed to pull image "****.azurecr.io/****:****": rpc error: code = Unknown desc = Error response from daemon: Get https://****.azurecr.io/v2/****/manifests/****: unauthorized: authentication required

So now I want to find the original client secret that the Service Principal uses, so that I can re-add that as a key to the Service Principal. That's the only solution I can think of other than recreating the entire cluster.

Any ideas?

Answer 1

Whoever comes over this issue there's an updated solution from Microsoft

https://docs.microsoft.com/en-us/azure/aks/update-credentials#update-aks-cluster-with-new-credentials

They also mention (something that's not obvious) that: By default, AKS clusters are created with a service principal that has a one-year expiration time.

Also, As of Azure CLI 2.0.68, the --password parameter to create a service principal with a user-defined password is no longer supported to prevent the accidental use of weak passwords. so the initial solution to change the service principal password doesn't work anymore.

Answer 2

In the end the solution was quite simple.

• In the Azure portal, navigate to the resource group named MC_<resourcegroup>_<aksName>_<region>.
• Click one of the resources of the type "Virtual machine".
• Scroll down to "Run command"

• Choose "RunShellScript"
• Enter cat /etc/kubernetes/azure.json and click "Run"

The command will return the contents of the JSON file. The property you need is aadClientSecret

Answer 3

It's an annoying thing that you want to do. For your issue, you cannot pull the image without authentication.

First, you have to find out the service principal of your container registry. You can do this in the Azure portal and navigate to the registry panel, then you can find the service principal like this:

Or you can use the Azure CLI command to find the registry ID like this:

az acr show --resource-group groupName --name registryName --query id --output tsv

Then use the command to find the service principal ID like this:

az role assignment list --scope registryID

You can select the service principal which you want.

Then you would get all the secrets with the command kubectl get secrets and kubectl get secrets secretName -o yaml to get the token of the secret. Then analyze one-by-one to check if the username the same as the service principal ID. You can use tools such as JWT to analyze the secret token. The result will like this:

If the username the same as the service principal ID which you find, that is the secret you want. This step is a trouble. You should check the secret one-by-one or you will have a more great way to check them.

By the way, it seems that you can just see the password of the service principal once when you create it. The Azure will not show you again. But if you create the Kubernetes secret, the password is stored in it.

Answer 4

There is new functionality available in the Azure portal to look at the cluster configuration without using the CLI. Go to the Azure Portal -> Your Cluster Resource -> Overview

On the right hand side there is a button that says "JSON View". Click it and you will see a json file with the cluster details. The service principal ID is visible under: "servicePrincipalProfile".