Does sharing the service principal for multiple AKS clusters allow one cluster to mess with the resources of another AKS cluster? Should one aim to create one service principal per AKS cluster? This practice seems to be encouraged both by the Azure CLI and tools such as Terraform.
Long story below:
We need to create one AKS cluster per customer to ensure another level of data isolation. Specifically, assume an attacker could somehow break into a Pod, escape Pod isolation, get access to a cluster's Kubernetes API and to the cluster's Service Principal (SP), said attacker should not get access to another cluster's persistent volume claims.
The Azure CLI creates a service principal once and reuses it for the next clusters (documentation, source code):
When you use the az aks create command to generate the service principal automatically, the service principal credentials are written to the file ~/.azure/aksServicePrincipal.json on the machine used to run the command.
Not being sure how much the cluster's SP can (or should) do, I found the following rather confusing information:
The documentation says:
To interact with Azure APIs, an AKS cluster requires an Azure Active Directory (AD) service principal. The service principal is needed to dynamically create and manage other Azure resources such as an Azure load balancer or container registry (ACR). [highlight mine]
This suggests that the cluster SP must have access to the cluster's Resource Group (RG) -- i.e., the RG where all the resource created by the cluster are located, not the RG in which the cluster is created. This is bad, since it means that sharing an SP across AKS clusters requires said SP to have access to all cluster RGs.
Later the documentation suggests that an SP without any role is sufficient:
az ad sp create-for-rbac --skip-assignment --name terraform-aks-ng-spI could also confirm this by checking from the Cloud Shell:
PS Azure:\> az role assignment list --assignee http://terraform-aks-ng-sp [] Azure:/The empty list shows that the SP has no roles. This is (or rather would be) good, since the share SP can be given access only to truly shared resources (e.g., a container registry).
However, the Azure Portal suggests that the SP does have access to the cluster's RG:
This feels like back to square 1. This is bad, since sharing an SP seems to implicitly give access to all cluster RGs.
Update: Sharing the SP among AKS clusters seems to be encouraged even by Terraform.
