Terraform azurerm_kubernetes_cluster default_node_pool orchestrator_version not updating

Question

I'm attempting to upgrade the default node pool k8s version using Terraform's azurerm_kubernetes_cluster, but it's telling me there is nothing to change. Here are the relevant details:

tfstate file (redacted, pulled from Terraform Cloud):

{
  "version": 4,
  "terraform_version": "1.0.2",
  "resources": [
    {
      "mode": "managed",
      "type": "azurerm_kubernetes_cluster",
      "name": "aks",
      "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "default_node_pool": [
              {
                "orchestrator_version": "1.19.7", // tfstate shows 1.19.7
                "type": "VirtualMachineScaleSets"
              }
            ],
            "kubernetes_version": "1.20.7"
          }
        }
      ]
    }
  ]
}

Resource definition:

resource "azurerm_kubernetes_cluster" "aks" {
  .
  .
  .
  kubernetes_version  = local.aks.kubernetes_version

  default_node_pool {
    name       = local.aks.default_node_pool.name
    vm_size    = local.aks.default_node_pool.vm_size
    node_count = local.aks.default_node_pool.node_count
    orchestrator_version = local.aks.default_node_pool.orchestrator_version
  }
  .
  .
  .
}

Variables:

variable "aks" {
    description = "Kubernetes Service values"
    type = object({
        name = string
        dns_prefix = optional(string)
        kubernetes_version = optional(string)
        default_node_pool = object({
            name = optional(string)
            vm_size = optional(string)
            node_count = optional(number)
            orchestrator_version = optional(string)
        })
        identity_type = optional(string)
    })
}



locals {
    aks = defaults(var.aks, {
        dns_prefix = var.aks.name
        kubernetes_version = "1.20.7"
        default_node_pool = {
            name = "agentpool"
            vm_size = "Standard_D4s_v3"
            node_count = 3
            orchestrator_version = "1.20.7"
        }
    })
}

Specifying kubernetes_version worked to upgrade the k8s control plan within Azure, but the default node pool did not update. After a bit more research I found the orchestrator_version, and to my understanding this should upgrade the k8s version for the virtual machine scale set.

When I run terraform plan, it tells me "Your infrastructure matches the configuration", even though tfstate clearly shows orchestrator_version as "1.19.7", and my variable (I've tried hard coding as well) is set to "1.20.7".

Why isn't Terraform / azurerm_kubernetes_cluster recognizing the orchestrator_version change?

The defaults function is part of an experimental feature even in 1.0. Have you enabled that feature anywhere? It is not present in your config in your question. — Matt Schuchard
I have enabled it, yes. I just didn't include it here for brevity — DoubleTK

Srijit_Bose-MSFT Srijit_Bose-MSFT · Accepted Answer · 2021-08-26T13:35:23

Can you please check using Terraform azurerm provider version 2.14.0? This release has a fix for this issue. Reference

A known workaround is to use a null_resource with a trigger on the Kubernetes version and running a Bash script with the local-exec provisioner. So, you do not have to upgrade the node pools manually.

...
resource "null_resource" "aks" {
  triggers = {
    aks_kubernetes_version = azurerm_kubernetes_cluster.aks.kubernetes_version
  }
  provisioner "local-exec" {
    command     = "./cluster-upgrade-fix.sh ${var.name} ${azurerm_resource_group.aks.name}"
    working_dir = path.module
  }
}
...

The script compares the Kubernetes version of the control plane and node pools. Differs the version it upgrades each node pool in the cluster accordingly.

cluster-upgrade-fix.sh :

#!/bin/bash
CLUSTER_NAME=$1
RESOURCE_GROUP=$2
CLUSTER_VERSION=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP | jq -r .kubernetesVersion)
NODE_POOLS=$(az aks nodepool list --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query '[].name' -o tsv)
for NODE_POOL in $NODE_POOLS; do
    NODE_VERSION=$(az aks nodepool show --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name $NODE_POOL | jq -r .orchestratorVersion)
    if [[ $CLUSTER_VERSION != $NODE_VERSION ]]; then
        az aks nodepool upgrade --kubernetes-version $CLUSTER_VERSION --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name $NODE_POOL --verbose
    fi
done

References:

Terraform azurerm_kubernetes_cluster default_node_pool orchestrator_version not updating

1 Answers