I can't make Kubernetes in AKS to pull the images from a private docker registry in Azure

Question

I have followed the instructions here https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret

So, I created a service principal

az ad sp create-for-rbac -n TerraformSP --role="Contributor" --scopes="/subscriptions/SUBSCRIPTION_ID"

This is the (resumed) terraform script

resource "azurerm_resource_group" "default" {
  name     = "myaks-rg"
  location = var.location
}

resource "azurerm_kubernetes_cluster" "default" {
  name                = "myaks"
  location            = azurerm_resource_group.default.location
  resource_group_name = azurerm_resource_group.default.name
  dns_prefix          = var.dns_prefix

  identity {
    type = "SystemAssigned"
  }

  role_based_access_control {
    enabled = true
  }
}

data "azurerm_container_registry" "acr" {
  name                = "myregistry"
  resource_group_name = "another-rg"
}

resource "azurerm_role_assignment" "aks_acr" {
  scope                            = data.azurerm_container_registry.acr.id
  role_definition_name             = "AcrPull"
  principal_id                     = azurerm_kubernetes_cluster.default.kubelet_identity[0].object_id
  skip_service_principal_aad_check = true
}

Then I executed terraform (using the IDs/Secret generated by az ad sp create-for-rbac command)

az logout

export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"
terraform apply

The AKS is created but the Pod cannot pull the images. This is error message in the pod.

Failed to pull image "myregistry.azurecr.io/myimage:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.azurecr.io/v2/myimage/manifests/latest: unauthorized: authentication required...

However, It works when I run terraform apply using my personal account (az login) and I don't set the environment variables ARM_CLIENT_ID, ARM_CLIENT_SECRET, etc...

I understand that the service principal TerraformSP (role Contributor) can't assign the role AcrPull to azurerm_kubernetes_cluster.default.kubelet_identity[0].object_id.

What is the best practice in this case? Should I assign some admin role to TerraformSP? Or, am I missing something in the terraform script?

The goal is to put this terraform script in a pipeline in Azure DevOps.

Cleyton T. Cleyton T. · Accepted Answer · 2020-12-03T15:24:08

I managed to make Kubernetes to pull images from my registry by re-creating the Service Principal with the role Owner.

az ad sp create-for-rbac -n TerraformSP --role="Owner" --scopes="/subscriptions/SUBSCRIPTION_ID"

I also noticed that terraform creates successfully the role assignment.

azurerm_role_assignment.aks_acr: Creation complete after 24s [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/another-rg/providers/Microsoft.ContainerRegistry/registries/myregistry/providers/Microsoft.Authorization/roleAssignments/SOME_ID]

I can't make Kubernetes in AKS to pull the images from a private docker registry in Azure

1 Answers