AWS certificate resolver for Traefik Ingress Controller in K3S Kubernetes Cluster with existing AWS HTTPS Load Balancer

Question

I have AWS K3S Kubernetes Cluster
I have AWS Load Balancer
I have registered domain
I have registered AWS Certificate
I created CNAME record for my domain and for AWS Load Balancer DNS name
I installed Traefik Ingress Controller on AWS K3S Kubernetes Cluster
I deployed "usermgmt" and "whoami" services to AWS K3S Kubernetes Cluster
I created Traefik Ingress with paths to "usermgmt" and "whoami"

The question is:

How to connect my AWS Load Balancer, which is hosted on my domain, to my services on K3s, using Ingress Traefik Controller?

Or in other words:

How to adapt "traefik service" or "traefik deployment", described below, to use AWS Certificate Resolver for my registered domain?

Or any example of how to use

AWS Load Balancer, AWS Target Group, AWS Security Group, created with Terraform files

in combination with Traefik Ingress Controller and Traefik Ingress Routes, deployed to K3S Kubernetes Cluster, resolved with AWS Certificate.

I currently can't connect to my services through AWS Load Balancer. The following errors are returned:

404 Page Not Found

502 Bad Gateway

Here are the examples of URLs, which I try:

https://keycloak.skycomposer.net/usermgmt
https://keycloak.skycomposer.net/whoami

I set up correspondent Ingress Routes for "usermgmt" and "whoami" kubernetes services.

Here is some more information:

I created K3S Kubernetes Cluster in AWS with Load Balancer

These are my terraform files: https://github.com/skyglass/user-management/tree/master/terraform

K3S cluster is deployed to EC2 instance (see "userdata.tpl" script)

I disabled Traefik Ingress Controller deployment, so I could deploy it later.

I found example on how to install "Traefik" to K3S Kubernetes cluster here: https://github.com/sleighzy/k3s-traefik-v2-kubernetes-crd

Unfortunately, this example uses "godaddy" certificate resolver, but my domain is registered with AWS Route 53 and I use AWS certificate manager.

Here are files for "traefik service" and "traefik deployment", which I try to adapt:

traefik-service:

---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: kube-system

spec:
  # The targetPort entries are required as the Traefik container is listening on ports > 1024
  # so that the container can be run as a non-root user and they can bind to these ports.
  # Traefik is still accessed over 80 and 443 on the host, but the service routes the traffic
  # to ports 8080 and 8443 on the container.
  ports:
    - protocol: TCP
      name: web
      port: 80
      targetPort: 8080
    - protocol: TCP
      name: websecure
      port: 443
      targetPort: 8443
    - protocol: TCP
      name: admin
      port: 8080
      targetPort: 9080
  selector:
    app: traefik
  # Set externalTrafficPolicy to Local so that all external traffic intended for
  # the Traefik pod goes directly to that local node. If the default of Cluster is
  # used instead then the client source IP address is lost, and may hop between nodes.
  externalTrafficPolicy: Local
  type: LoadBalancer

traefik-deployment:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-system
  name: traefik-ingress-controller

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: kube-system
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.4
          args:
            - --api.dashboard=true
            - --ping=true
            - --accesslog
            - --entrypoints.traefik.address=:9080
            - --entrypoints.web.address=:8080
            - --entrypoints.websecure.address=:8443
            # Uncomment the below lines to redirect http requests to https.
            # This specifies the port :443 and not the https entrypoint name for the
            # redirect as the service is listening on port 443 and directing traffic
            # to the 8443 target port. If the entrypoint name "websecure" was used,
            # instead of "to=:443", then the browser would be redirected to port 8443.
            - --entrypoints.web.http.redirections.entrypoint.to=:443
            - --entrypoints.web.http.redirections.entrypoint.scheme=https
            - --providers.kubernetescrd
            - --providers.kubernetesingress
            - --certificatesresolvers.myresolver.acme.tlschallenge=true
            - --certificatesresolvers.myresolver.acme.email=postmaster@example.com
            - --certificatesresolvers.myresolver.acme.storage=/etc/traefik/certs/acme.json          
            # Please note that this is the staging Let's Encrypt server.
            # Once you get things working, you should remove that whole line altogether.
            # - --certificatesresolvers.godaddy.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
            - --log
            - --log.level=INFO
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /ping
              port: 9080
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
          resources:
            limits:
              memory: '100Mi'
              cpu: '1000m'
          ports:
            # The Traefik container is listening on ports > 1024 so the container
            # can be run as a non-root user and they can bind to these ports.
            - name: web
              containerPort: 8080
            - name: websecure
              containerPort: 8443
            - name: admin
              containerPort: 9080
          volumeMounts:
            - name: certificates
              mountPath: /etc/traefik/certs
      # volumes:
      #   - name: certificates
      #     persistentVolumeClaim:
      #       claimName: traefik-certs-pvc              
      volumes:
        - name: certificates
          hostPath:
            path: "/Users/dddd/git/aws/letsencrypt:/etc/traefik/certs"

See other files here: https://github.com/sleighzy/k3s-traefik-v2-kubernetes-crd

Ideally there should be solution like this:

apiVersion: v1
kind: Service
metadata:
  name: traefik-proxy
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:REGION:ACCOUNTID:certificate/CERT-ID"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
spec:
  type: LoadBalancer
  selector:
    app: traefik-proxy
    tier: proxy
  ports:
  - port: 443
    targetPort: 80

In this solution, I would just provide my AWS Certificate ARN and traefik ingress controller will do everything else.

The similar solution is described in this article:

https://www.ronaldjamesgroup.com/blog/getting-started-with-traefik

But, unfortunately, this solution doesn't work for me too, I tried it without any success.

The following errors are returned:

404 Page Not Found

502 Bad Gateway

when I try Ingress Route Paths for my domain:

https://keycloak.skycomposer.net/usermgmt
https://keycloak.skycomposer.net/whoami

Mykhailo Skliar Mykhailo Skliar · Accepted Answer · 2021-06-01T01:31:15

After trying several options, I finally found the solution: https://github.com/skyglass-examples/aws-k3s-traefik

I created AWS Load Balancer and K3S cluster with Terraform
I created Traefik Ingress Controller kubernetes manifest files
I created kubernetes manifest files for 2 services
I registered AWS Load Balancer DNS name for my domain
I created AWS Certificate for my domain
I used AWS Certificate ARN for Traefik Ingress Controller and AWS HTTPS Load Balancer

Here are my Traefik Ingress Controller manifest files:

traefik-deployment.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system


---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik-proxy
  namespace: kube-system     
  labels:
    app: traefik-proxy
    tier: proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-proxy
      tier: proxy
  template:
    metadata:
      labels:
        app: traefik-proxy
        tier: proxy
    spec:
      serviceAccountName: traefik-ingress-controller   
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:v1.2.0-rc1-alpine
        name: traefik-proxy
        ports:
        - containerPort: 80
          hostPort: 80
          name: traefik-proxy
        - containerPort: 8080
          name: traefik-ui
        args:
          - --web
          - --kubernetes

traefik-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: traefik-proxy
  namespace: kube-system   
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-1:dddddddddd"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
    service.beta.kubernetes.io/aws-load-balancer-type: "alb"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local  
  selector:
    app: traefik-proxy
    tier: proxy
  ports:
  - port: 443
    targetPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system     
spec:
  selector:
    app: traefik-proxy
    tier: proxy
  ports:
  - port: 80
    targetPort: 8080

traefik-ingress.yaml:

apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
  name: traefik-lb
spec:
  controller: traefik.io/ingress-controller

---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "traefik-usermgmt-ingress"
spec:
  ingressClassName: "traefik-lb"
  rules:
  - host: "keycloak.skycomposer.net"
    http:
      paths:
      - path: "/usermgmt"
        backend:
          serviceName: "usermgmt"
          servicePort: 80


---
apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "traefik-whoami-ingress"
spec:
  ingressClassName: "traefik-lb"
  rules:
  - host: "keycloak.skycomposer.net"
    http:
      paths:
      - path: "/whoami"
        backend:
          serviceName: "whoami"
          servicePort: 80

See the full code here: https://github.com/skyglass-examples/aws-k3s-traefik

The code includes:

terraform files for AWS Load Balancer and K3S Kubernetes Cluster
source code for one of the docker containers, which I deployed to K3S
kubernetes manifest files for Traefik Ingress Controller, 2 Kubernetes Services and Traefik Ingress, which exposes these services with secured HTTPS connection on registered domain.
Replace AWS Certificate ARN with correspondent ARN of your certificate
Replace "skycomposer.net" with your domain name (see more details in the Readme file: https://github.com/skyglass-examples/aws-k3s-traefik)

AWS certificate resolver for Traefik Ingress Controller in K3S Kubernetes Cluster with existing AWS HTTPS Load Balancer

1 Answers