Use Kubernetes provider on Terraform with private GKE cluster

Question

I want to use the kubernetes provider on Terraform to interact with a private GKE cluster. I can successfully create the cluster however I am unable to create the namespace, I consistently get the timeout error. Authentication is not as issue as I can run kubectl ... commands locally without issue. I believe the issue relates to the fact the cluster is private (as all the examples I found relate to public clusters). Does anyone know how to connect the kubernetes provider to a private GKE cluster?

My main.tf file:

provider "google" {
 project = "<PROJECT_ID>"
}

variable "cluster_name" {
  default = "<CLUSTER_NAME>"
}

resource "google_container_cluster" "composer_cluster" {
  name      = var.cluster_name
  location  = "europe-west1-b"

  # Node
  initial_node_count = 1
  node_config {
    disk_size_gb  = 100
    disk_type     = "pd-standard"
    machine_type  = "n1-standard-4"
    metadata      = {
      disable-legacy-endpoints= true
    }
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
    service_account = "<SERVICE_ACCOUNT>"
  }

  # Network
  network     = "<NETWORK>"
  subnetwork  = "<SUBNETWORK>"

  # IP allocation
  private_cluster_config {
    enable_private_endpoint= true
    enable_private_nodes= true
    master_global_access_config {
      enabled= true
    }
    master_ipv4_cidr_block= "172.16.32.0/28"
  }
  ip_allocation_policy {
    cluster_ipv4_cidr_block= "10.92.0.0/14"
    services_ipv4_cidr_block= "10.82.240.0/20"
  }

  # Security
  enable_kubernetes_alpha= false
  enable_legacy_abac= false
  enable_intranode_visibility= true
  master_authorized_networks_config {}
  network_policy {
    enabled= true
    provider= "CALICO"
  }
  enable_shielded_nodes= true

  # Timeouts
  timeouts {
    create = "30m"
    update = "40m"
  }
}

data "google_client_config" "current" {}

provider "kubernetes" {
  host    = google_container_cluster.composer_cluster.private_cluster_config[0].private_endpoint
  token   = data.google_client_config.current.access_token
  client_certificate = base64decode(google_container_cluster.composer_cluster.master_auth[0].client_certificate)
  client_key = base64decode(google_container_cluster.composer_cluster.master_auth[0].client_key)
  cluster_ca_certificate = base64decode(google_container_cluster.composer_cluster.master_auth[0].cluster_ca_certificate)
}

resource "null_resource" "get-credentials" {
 depends_on = [google_container_cluster.composer_cluster] 
 provisioner "local-exec" {
   command = "gcloud container clusters get-credentials ${google_container_cluster.composer_cluster.name} --internal-ip --zone europe-west1-b --project <PROJECT_ID>"
 }
}

resource "kubernetes_namespace" "namespace" {
  metadata {
    labels = {
      app = "create-namespace"
    }
    name = "<NAMESPACE>"
  }
  depends_on = [null_resource.get-credentials]
}

Output:

oogle_container_cluster.composer_cluster: Creating...
google_container_cluster.composer_cluster: Still creating... [10s elapsed]
google_container_cluster.composer_cluster: Still creating... [20s elapsed]
google_container_cluster.composer_cluster: Still creating... [30s elapsed]
google_container_cluster.composer_cluster: Still creating... [40s elapsed]
google_container_cluster.composer_cluster: Still creating... [50s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [5m0s elapsed]
google_container_cluster.composer_cluster: Creation complete after 5m2s [id=projects/<PROJECT_ID>/locations/europe-west1-b/clusters/<CLUSTER_NAME>]
kubernetes_namespace.namespace: Creating...
kubernetes_namespace.namespace: Still creating... [10s elapsed]
kubernetes_namespace.namespace: Still creating... [20s elapsed]
kubernetes_namespace.namespace: Still creating... [30s elapsed]

Error: Post "https://172.16.32.2/api/v1/namespaces": dial tcp 172.16.32.2:443: i/o timeout

Georges Lorré Georges Lorré · Accepted Answer · 2021-02-23T13:45:58

Still seems like an authentication issue, I've had succes with running the gcloud module to authenticate to a GKE cluster:

module "gcloud" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 0.5"

  platform = "linux"

  create_cmd_entrypoint  = "gcloud"
  create_cmd_body        = "container clusters get-credentials ${google_container_cluster.composer_cluster.name} --region=${var.zone}"
}

provider "kubernetes" {
# the authorization is handled by running gcloud clusters get-credentials using the gcloud terraform module
}

resource "kubernetes_deployment" "main" {
....
}

Use Kubernetes provider on Terraform with private GKE cluster

1 Answers