AWS EKS terraform tutorial (with assumeRole) - k8s dashboard error

Question

I followed the tutorial at https://learn.hashicorp.com/tutorials/terraform/eks. Everything works fine with a single IAM user with the required permissions as specified at https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/iam-permissions.md

But when I try to assumeRole in a cross AWSAccount scenario I run into errors/failures.

I started kubectl proxy as per step 5.

However, when I try to access the k8s dashboard at http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ (after completing steps 1-5), I get the error message as follows -

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "no endpoints available for service \"kubernetes-dashboard\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

I also got zero pods in READY state for the metrics server deployment in step 3 of the tutorial -

$ kubectl get deployment metrics-server -n kube-system
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   0/1     1            0           21m

My kube dns too has zero pods in READY state and the status is -

kubectl -n kube-system -l=k8s-app=kube-dns get pod
NAME                       READY   STATUS    RESTARTS   AGE
coredns-55cbf8d6c5-5h8md   0/1     Pending   0          10m
coredns-55cbf8d6c5-n7wp8   0/1     Pending   0          10m

My terraform version info is as below -

$ terraform version
2021/03/06 21:18:18 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
  Use TF_LOG=TRACE to see Terraform's internal logs.
  ----
2021/03/06 21:18:18 [INFO] Terraform version: 0.14.7  
2021/03/06 21:18:18 [INFO] Go runtime version: go1.15.6
2021/03/06 21:18:18 [INFO] CLI args: []string{"/usr/local/bin/terraform", "version"}
2021/03/06 21:18:18 [DEBUG] Attempting to open CLI config file: /Users/user1/.terraformrc
2021/03/06 21:18:18 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021/03/06 21:18:18 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021/03/06 21:18:18 [DEBUG] ignoring non-existing provider search directory /Users/user1/.terraform.d/plugins
2021/03/06 21:18:18 [DEBUG] ignoring non-existing provider search directory /Users/user1/Library/Application Support/io.terraform/plugins
2021/03/06 21:18:18 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021/03/06 21:18:18 [INFO] CLI command args: []string{"version"}
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/aws v3.31.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.2
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Output of describe pods for kube-system ns is -

$ kubectl describe pods -n kube-system
Name:               coredns-7dcf49c5dd-kffzw
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               <none>
Labels:             eks.amazonaws.com/component=coredns
                    k8s-app=kube-dns
                    pod-template-hash=7dcf49c5dd
Annotations:        eks.amazonaws.com/compute-type: ec2
                    kubernetes.io/psp: eks.privileged
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/coredns-7dcf49c5dd
Containers:
  coredns:
    Image:       602401143452.dkr.ecr.ca-central-1.amazonaws.com/eks/coredns:v1.8.0-eksbuild.1
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-sqv8j (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-sqv8j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-sqv8j
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  34s (x16 over 15m)  default-scheduler  no nodes available to schedule pods


Name:               coredns-7dcf49c5dd-rdw94
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               <none>
Labels:             eks.amazonaws.com/component=coredns
                    k8s-app=kube-dns
                    pod-template-hash=7dcf49c5dd
Annotations:        eks.amazonaws.com/compute-type: ec2
                    kubernetes.io/psp: eks.privileged
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/coredns-7dcf49c5dd
Containers:
  coredns:
    Image:       602401143452.dkr.ecr.ca-central-1.amazonaws.com/eks/coredns:v1.8.0-eksbuild.1
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-sqv8j (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-sqv8j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-sqv8j
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  35s (x16 over 15m)  default-scheduler  no nodes available to schedule pods


Name:               metrics-server-5889d4b758-2bmc4
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             k8s-app=metrics-server
                    pod-template-hash=5889d4b758
Annotations:        kubernetes.io/psp: eks.privileged
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/metrics-server-5889d4b758
Containers:
  metrics-server:
    Image:        k8s.gcr.io/metrics-server-amd64:v0.3.6
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-wsqkn (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  metrics-server-token-wsqkn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-wsqkn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  6s (x9 over 6m56s)  default-scheduler  no nodes available to schedule pods

Also,

$ kubectl get nodes
No resources found.

And,

$ kubectl describe nodes

returns nothing

Can someone help me troubleshoot and fix this ?

TIA.

What do you see if you describe the pods? You should get a hint re why they are not starting and left in pending. — mreferre
Updated OP with details as character limit hit for comments.. TLDR -> 'no nodes available to schedule pods' — ZeroGraviti
Apparently what you did is to deploy an EKS control plane without any node to support workloads. When you deploy EKS you (typically) need to deploy a control plane and a data plane. The data plane can be self-managed EC2 instances, managed EKS node groups or Fargate. If you are just getting started my suggestion is to use eksctl as described here to deploy a cluster with a managed node group. — mreferre
Ouch. Sorry I completely overlooked your references to Terraform. The tutorial you are following does indeed create nodes. My suggestion would be to troubleshoot first why those nodes are not deployed and/or why they are not joining the cluster. I would start to see if the corresponding EC2 instances are actually deployed and work from there to figure out why Terraform wasn't able to make them join the cluster. — mreferre

ZeroGraviti ZeroGraviti · Accepted Answer · 2021-04-25T03:27:13

Self documenting my solution

Given my AWS setup is as follows account1:user1:role1 account2:user2:role2

and the role setup is as below - arn:aws:iam::account2:role/role2 << trust relationship >>

eks.amazonaws.com
ec2.amazonaws.com
arn:aws:iam::account1:user/user1
arn:aws:sts::account2:assumed-role/role2/user11

Updating the eks-cluster.tf as below -

map_roles = [ 
    {
      "groups": [ "system:masters" ], 
      "rolearn": "arn:aws:iam::account2:role/role2", 
      "username": "role2"
    }
  ]
  map_users = [ 
    { 
      "groups": [ "system:masters" ], 
      "userarn": "arn:aws:iam::account1:user/user1", 
      "username": "user1"
    }, 
   { 
      "groups": [ "system:masters" ], 
      "userarn": "arn:aws:sts::account2:assumed-role/role2/user11", 
      "username": "user1"
    } 
  ]

p.s.: Yes "user11" is a generated username suffixed with a "1" to the account1 user with a username of "user1"

Makes everything work !

AWS EKS terraform tutorial (with assumeRole) - k8s dashboard error

1 Answers