Using MFA with EKS kubectl & aws-iam-authenticator

Question

I've been trying to get MFA working with kubectl to secure access to the EKS masters in AWS. Docs seem to hint that it's possible but I'm running into problems and I can't figure it out.

Without anything special, I can connect to my EKS cluster:

kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   2d

If I add MFA to the AWS CLI, following these posts:

How to use MFA with AWS CLI?

https://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html

I set up my local credentials like so:

[default]
aws_access_key_id = **********************
aws_secret_access_key = **********************
region = us-west-2

[default_role]
mfa_serial = arn:aws:iam::1234567890:mfa/spanktar
role_arn = arn:aws:iam::1234567890:role/test_assumeRole
source_profile = default

...then I can see that the AWS CLI has MFA:

aws sts get-caller-identity --profile default_role
Enter MFA code for arn:aws:iam::1234567890:mfa/spanktar: 123456
{
    "UserId": "**********************:botocore-session-1234567890",
    "Account": "1234567890",
    "Arn": "arn:aws:sts::1234567890:assumed-role/test_assumeRole/botocore-session-1234567890"
}

Then I can verify:

aws sts assume-role --role-arn arn:aws:iam::1234567890:role/test_assumeRole --role-session-name default_role
{
    "Credentials": {
        "AccessKeyId": "**********************",
        "SecretAccessKey": "**********************",
        "SessionToken": "FOO",
        "Expiration": "2018-10-11T21:19:20Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "**********************:default_role",
        "Arn": "arn:aws:sts::1234567890:assumed-role/test_assumeRole/default_role"
    }
}

kubectl still works as normal because it isn't using the profile yet:

kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   2d

Then I set up k8s to use the profile, following this documentation

users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: "AWS_PROFILE"
        value: "default_role"
      args:
        - "token"
        - "-i"
        - "test-eks-cluster"

So now when I try kubectl it asks for MFA, but it can never be satisfied:

kubectl get svc
Assume Role MFA token code: 123456
Assume Role MFA token code: 123456
could not get token: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code.
    status code: 403, request id: 123456-cd93-11e8-80a5-1234567898765
E1011 13:22:07.385200   28191 exec.go:230] refreshing credentials: exec: exit status 1
No resources found.
error: You must be logged in to the server (Unauthorized)

The AWS Role test_assumeRole looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::1234567890:user/spanktar"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The related policy test_assumePolicy looks like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::1234567890:role/test_assumeRole"
        }
    ]
}

So, the questions:

Do I need to attach any of this to the IAM user? The docs seem to suggest not.
What am I missing here to get this to operate? It seems almost wired up correctly.

Hi did you manage to solve this ? I am currently facing the exact same problem and tearing my hair out trying to work out what's wrong ! Thanks ! — user10673977

Hendric Hendric · Accepted Answer · 2019-01-22T13:12:20

I had the same issue and managed it by adding the -r option to the kubectl config file.

In your case it would look like this:

users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: "AWS_PROFILE"
        value: "default_role"
      args:
        - "token"
        - "-i"
        - "test-eks-cluster"
        - "-r"
        - "arn:aws:iam::1234567890:role/test_assumeRole"

I hope this works for you too.

Using MFA with EKS kubectl & aws-iam-authenticator

1 Answers