Kubernetes Service Account and AWS IAM Role for Secrets

Question

I have a setup of EKS Cluster and I am trying to access some of the secrets available in my AWS Secrets manager, Currently i have given permissions to one ROLE (AWS-IAM) to access all the required secrets and i am using below k8s manifest and i am getting below error.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark-pi
  namespace: spark-pi
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXX:role/spark-secret-role
automountServiceAccountToken: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spark-pi-role
  namespace: spark-pi  
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spark-pi-role-binding
  namespace: spark-pi
subjects:
- kind: ServiceAccount
  name: spark-pi
  namespace: spark-pi
roleRef:
  kind: Role
  name: spark-pi-role
  apiGroup: rbac.authorization.k8s.io

I am able to Submit Spark Job Successfully, but i am getting below Error while checking POD logs. User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/eks-node-role/i-0XXXXXXXXXXX is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-west-2:XXXXXXXXXX:secret:dev/somesecret (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException;

Not Sure why it is assuming EKS-Node Role, when i have attached Required Role and Permissions to Service Account, also i have already created Managed Policy (Attached to Role) to access AWS Secrets.

Did you take a look forums.aws.amazon.com/thread.jspa?messageID=897702 docs.aws.amazon.com/IAM/latest/UserGuide/…? — Malgorzata
@Malgorzata Yes, but it does not seem to resolution in my case, you can direct to any point which i am missing — Zester07
Are you using specific tutorial for setup ? Please read and follow this instruction aws.amazon.com/blogs/containers/aws-secrets-controller-poc — Malgorzata

Julian Kleinhans Julian Kleinhans · Accepted Answer · 2021-01-24T17:38:20

RBAC itself is just for kubernetes access management. You have defined a ServiceAccount with an AWS role attached, thats good.

Could you share the AWS policy which is attached to the role role/spark-secret-role? And please share the Pod manifest with us, you need to attach the ServiceAccount to the Pod itself. Otherwise the Pod is not using the ServiceAccount with the attached AWS role.

You also need to create an OIDC ID provider (IdP) in AWS.

The whole thing is called IRSA (IAM Roles for Service Accounts) You can find all necessary information in this AWS blog article: https://aws.amazon.com/de/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

Kubernetes Service Account and AWS IAM Role for Secrets

1 Answers