Not able to join worker nodes using kubectl with updated aws-auth configmap

Question

I'm setting up AWS EKS cluster using terraform from an EC2 instance. Basically the setup includes EC2 launch configuration and autoscaling for worker nodes. After creating the cluster, I am able to configure kubectl with aws-iam-authenticator. When I did

kubectl get nodes

It returned

No resources found

as the worker nodes were not joined. So I tried updating aws-auth-cm.yaml file

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

with IAM role ARN of the worker node. And did

kubectl apply -f aws-auth-cm.yaml

It returned

ConfigMap/aws-auth created

Then I understood that role ARN configured in aws-auth-cm.yaml is the wrong one. So I updated the same file with the exact worker node role ARN.

But this time I got 403 when I did kubectl apply -f aws-auth-cm.yaml again.

It returned

Error from server (Forbidden): error when retrieving current configuration of: Resource: "/v1, Resource=configmaps", GroupVersionKind: "/v1, Kind=ConfigMap" Name: "aws-auth", Namespace: "kube-system" Object: &{map["apiVersion":"v1" "data":map["mapRoles":"- rolearn: arn:aws:iam::XXXXXXXXX:role/worker-node-role\n username: system:node:{{EC2PrivateDNSName}}\n groups:\n - system:bootstrappers\n - system:nodes\n"] "kind":"ConfigMap" "metadata":map["name":"aws-auth" "namespace":"kube-system" "annotations":map["kubectl.kubernetes.io/last-applied-configuration":""]]]} from server for: "/home/username/aws-auth-cm.yaml": configmaps "aws-auth" is forbidden: User "system:node:ip-XXX-XX-XX-XX.ec2.internal" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

I'm not able to reconfigure the ConfigMap after this step.

I'm getting 403 for commands like

kubectl apply
kubectl delete
kubectl edit

for configmaps. Any help?

I suppose that you must execute kubectl command with IAM user entitled with appropriate RBAC enabled. — Nick_Kh
@mk_sta kubectl is configured with correct IAM user role. Other commands like kubectl cluster-info, kubectl get nodes.., works fine — Magesh
You can check in RBAC whether this user is granted to read configmaps in kube-system namespace: kubectl auth can-i get configmaps -n kube-system — Nick_Kh

Magesh Magesh · Accepted Answer · 2019-06-19T09:28:07

I found the reason why kubectl returned 403 for this scenario.

As per this doc, the user/role who created the cluster will be given system:masters permissions in the cluster's RBAC configuration

When I tried to create a ConfigMap for aws-auth to join worker nodes, I gave the ARN of role/user who created the cluster instead of ARN of worker nodes.

And it updated the group(system:masters) of admin with groups system:bootstrappers and system:nodes in RBAC which basically locked the admin himself. And it is not recoverable since admin has lost the privileges from group system:masters.

Not able to join worker nodes using kubectl with updated aws-auth configmap

1 Answers