EKS kube-system deployments CrashLoopBackOff

Question

I am trying to deploy Kube State Metrics into the kube-system namespace in my EKS Cluster (eks.4) running Kubernetes v1.14.

Kubernetes Connection

provider "kubernetes" {
  host                   = var.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
  cluster_ca_certificate = base64decode(var.cluster.certificate)
  load_config_file       = true
}

Deployment Manifest (as .tf)

resource "kubernetes_deployment" "kube_state_metrics" {
  metadata {
    name      = "kube-state-metrics"
    namespace = "kube-system"

    labels = {
      k8s-app = "kube-state-metrics"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        k8s-app = "kube-state-metrics"
      }
    }

    template {
      metadata {
        labels = {
          k8s-app = "kube-state-metrics"
        }
      }

      spec {
        container {
          name  = "kube-state-metrics"
          image = "quay.io/coreos/kube-state-metrics:v1.7.2"

          port {
            name           = "http-metrics"
            container_port = 8080
          }

          port {
            name           = "telemetry"
            container_port = 8081
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }

          readiness_probe {
            http_get {
              path = "/"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }
        }

        service_account_name = "kube-state-metrics"
      }
    }
  }
}

I have deployed all the required RBAC manifests from https://github.com/kubernetes/kube-state-metrics/tree/master/kubernetes as well - redacted here for brevity.

When I run terraform apply on the deployment above, the Terraform output is as follows : kubernetes_deployment.kube_state_metrics: Still creating... [6m50s elapsed]

Eventually timing out at 10m.

Here are the outputs of the logs for the kube-state-metrics pod

I0910 23:41:19.412496       1 main.go:140] metric white-blacklisting: blacklisting the following items:
W0910 23:41:19.412535       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0910 23:41:19.412565       1 client_config.go:546] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
F0910 23:41:19.412782       1 main.go:148] Failed to create client: invalid configuration: no configuration has been provided

Is that the only thing logged? That gets logged a fair amount and is a general warning in the Kubernetes client but is probably not the thing that matters here. — ydaetskcoR

bxbxckx bxbxckx · Accepted Answer · 2019-09-10T23:58:27

Adding the following to the spec has taken me to a successful deployment.

automount_service_account_token = true

For posterity :

resource "kubernetes_deployment" "kube_state_metrics" {
  metadata {
    name      = "kube-state-metrics"
    namespace = "kube-system"

    labels = {
      k8s-app = "kube-state-metrics"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        k8s-app = "kube-state-metrics"
      }
    }

    template {
      metadata {
        labels = {
          k8s-app = "kube-state-metrics"
        }
      }

      spec {
        automount_service_account_token = true
        container {
          name  = "kube-state-metrics"
          image = "quay.io/coreos/kube-state-metrics:v1.7.2"

          port {
            name           = "http-metrics"
            container_port = 8080
          }

          port {
            name           = "telemetry"
            container_port = 8081
          }

          liveness_probe {
            http_get {
              path = "/healthz"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }

          readiness_probe {
            http_get {
              path = "/"
              port = "8080"
            }

            initial_delay_seconds = 5
            timeout_seconds       = 5
          }
        }

        service_account_name = "kube-state-metrics"
      }
    }
  }
}

EKS kube-system deployments CrashLoopBackOff

2 Answers