AWS ALB Ingress Controller with IRSA authorization error

Question

I am trying to setup an AWS ALB Ingress Controller using the IRSA method instead of kube2iam. There is however some lack of documentation so I came to a dead end.

What I did so far:

• Configured the OIDC provider for my cluster

eksctl utils associate-iam-oidc-provider --cluster devops --approve

• Created the proper policy by using the template

• Created the IAM service account that will be used by the Ingress Controller and associated the policy

eksctl create iamserviceaccount --name alb-ingress --namespace default --cluster devops --attach-policy-arn arn:aws:iam::112233445566:policy/eks-ingressController-iam-policy-IngressControllerPolicy-1111111111 --approve

• Deployed required rbac rules provided

kubectl apply -f rbac-role.yaml

• Deployed the AWS Ingress Controller by using this template. Payed attention so the ServiceAccount matches the service account I created previously.

Everything up to here is deployed fine. Now I try to deploy my Ingress service but I get this error (in the controller logs)

kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/eksctl-devops-nodegroup-ng-1-work-NodeInstanceRole-J08FDJHIWPI7/i-000000000000 is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: 94d614a1-c05d-4b92-8ad6-86b450407f6a"  "Controller"="alb-ingress-controller" "Request"={"Namespace":"superset","Name":"superset-ingress"}

Obviously the node doesn't have the proper permissions for the ALB creation, and I guess that if I attached my policy to the role stated in the log it would work. But that defeats the whole purpose of doing the IRSA method right?

What I would expect is for the Ingress Controller pod to need the appropriate permissions -by using the service account- to create the ALB and not the Node. Am I missing something here?

Answer 1

So, in case someone comes up to the same problem.

The solution is, when creating the rbac roles, to comment out from the rbac-role.yaml (as provided here) the last part which creates the service account.

Since we already created a service account with eksctl and attached to it the aws policy, we can attach to this service account the rbac permissions also. Then this service account can be used normally in the ingress controller pod to do its magic.

Answer 2

According to the documentation need the permission to CRUD an ALB. You could if you wanted to try giving just the ALB driver Pod a role with permissions create the ALB but I have not tested it and I am not sure it matters, if your entire scheduler has been given access to to use the ALB driver/pod to create these objects on AWS.

I am not using EKS's 3.0's cluster creation tool, instead I have my own CFT that I use to create workers due to my orgs additional security requirements.

I have have created and attached the bellow managed policy to workers that need to create ALB's and it just works.

  ALBPolicy:
    Type: "AWS::IAM::ManagedPolicy"
    Properties:
      Description: Allows workers to CRUD alb's
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Action:
              - "acm:DescribeCertificate"
              - "acm:ListCertificates"
              - "acm:GetCertificate"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "ec2:AuthorizeSecurityGroupIngress"
              - "ec2:CreateSecurityGroup"
              - "ec2:CreateTags"
              - "ec2:DeleteTags"
              - "ec2:DeleteSecurityGroup"
              - "ec2:DescribeAccountAttributes"
              - "ec2:DescribeAddresses"
              - "ec2:DescribeInstances"
              - "ec2:DescribeInstanceStatus"
              - "ec2:DescribeInternetGateways"
              - "ec2:DescribeNetworkInterfaces"
              - "ec2:DescribeSecurityGroups"
              - "ec2:DescribeSubnets"
              - "ec2:DescribeTags"
              - "ec2:DescribeVpcs"
              - "ec2:ModifyInstanceAttribute"
              - "ec2:ModifyNetworkInterfaceAttribute"
              - "ec2:RevokeSecurityGroupIngress"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "elasticloadbalancing:AddListenerCertificates"
              - "elasticloadbalancing:AddTags"
              - "elasticloadbalancing:CreateListener"
              - "elasticloadbalancing:CreateLoadBalancer"
              - "elasticloadbalancing:CreateRule"
              - "elasticloadbalancing:CreateTargetGroup"
              - "elasticloadbalancing:DeleteListener"
              - "elasticloadbalancing:DeleteLoadBalancer"
              - "elasticloadbalancing:DeleteRule"
              - "elasticloadbalancing:DeleteTargetGroup"
              - "elasticloadbalancing:DeregisterTargets"
              - "elasticloadbalancing:DescribeListenerCertificates"
              - "elasticloadbalancing:DescribeListeners"
              - "elasticloadbalancing:DescribeLoadBalancers"
              - "elasticloadbalancing:DescribeLoadBalancerAttributes"
              - "elasticloadbalancing:DescribeRules"
              - "elasticloadbalancing:DescribeSSLPolicies"
              - "elasticloadbalancing:DescribeTags"
              - "elasticloadbalancing:DescribeTargetGroups"
              - "elasticloadbalancing:DescribeTargetGroupAttributes"
              - "elasticloadbalancing:DescribeTargetHealth"
              - "elasticloadbalancing:ModifyListener"
              - "elasticloadbalancing:ModifyLoadBalancerAttributes"
              - "elasticloadbalancing:ModifyRule"
              - "elasticloadbalancing:ModifyTargetGroup"
              - "elasticloadbalancing:ModifyTargetGroupAttributes"
              - "elasticloadbalancing:RegisterTargets"
              - "elasticloadbalancing:RemoveListenerCertificates"
              - "elasticloadbalancing:RemoveTags"
              - "elasticloadbalancing:SetIpAddressType"
              - "elasticloadbalancing:SetSecurityGroups"
              - "elasticloadbalancing:SetSubnets"
              - "elasticloadbalancing:SetWebACL"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "iam:CreateServiceLinkedRole"
              - "iam:GetServerCertificate"
              - "iam:ListServerCertificates"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "cognito-idp:DescribeUserPoolClient"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "waf-regional:GetWebACLForResource"
              - "waf-regional:GetWebACL"
              - "waf-regional:AssociateWebACL"
              - "waf-regional:DisassociateWebACL"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "tag:GetResources"
              - "tag:TagResources"
            Resource: "*"
          -
            Effect: "Allow"
            Action:
              - "waf:GetWebACL"
            Resource: "*"

Answer 3

I've got a similar error (not identical) when using version v1.1.8 of this controller:

kubebuilder/controller "msg"="Reconciler
error"="failed get WAFv2 webACL for load balancer arn:aws:elasticloadbalancing:...: AccessDeniedException: User: arn:aws:sts:::assumed-role/eks-node-group-role/ is not authorized to perform: wafv2:GetWebACLForResource on resource: arn:aws:wafv2:us-east-2::regional/webacl/*\n\tstatus code: 400, request id: ..."
"controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"aws-alb-ingress"}

I'll add it because I think it can help people which search under the same error message.

The reason for the error described above was the fact that version v1.1.7 of this controller needs new IAM permissions in the nodegroup role's *PolicyALBIngress policy.

(!) Be aware that the new IAM permission is required even no wafv2 annotation is used.

Solution 1

Adding the section of wafv2 allow actions to the policy:

{
  "Effect": "Allow",
  "Action": [
    "wafv2:GetWebACL",
    "wafv2:GetWebACLForResource",
    "wafv2:AssociateWebACL",
    "wafv2:DisassociateWebACL"
  ],
  "Resource": "*"
}

Solution 2

WAFV2 support can be disabled by controller flags as mentioned here.

A) If you install it via kubectl, add - --feature-gates=waf=false to the spec -> containers -> args section.

B) If you install it via helm, add --set extraArgs."feature-gates"='waf=false' in helm upgrade command.

---

Notice that this requirment was already being updated in the eksctl tool (Review also in here).

---

Additional reference.