How to choose ingress controller in values.yaml

Question

Context

I have a k3s (v1.19.3+k3s3) cluster and I am using the 'out of the box' traefik ingress controler

kubectl  get svc -n kube-system 
NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
kube-dns             ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       5d16h
metrics-server       ClusterIP      10.43.127.152   <none>          443/TCP                      5d16h
traefik-prometheus   ClusterIP      10.43.73.131    <none>          9100/TCP                     5d16h
traefik              LoadBalancer   10.43.121.125   111.11.11.111   80:32492/TCP,443:31586/TCP   5d16h

kubectl   -n kube-system describe svc traefik
Name:                     traefik
Namespace:                kube-system
Labels:                   app=traefik
                          app.kubernetes.io/managed-by=Helm
                          chart=traefik-1.81.0
                          heritage=Helm
                          release=traefik
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app=traefik,release=traefik
Type:                     LoadBalancer
IP:                       10.43.121.125
LoadBalancer Ingress:     111.11.11.111
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32492/TCP
Endpoints:                10.42.0.6:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31586/TCP
Endpoints:                10.42.0.6:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

The issue

I do not find how to use this ingress controller using a helm chart (say, this ghost chart). In my understanding of how Helm charts work and the purpose they fulfill, Ingress templating allow to use different ingress controllers. Here is the chart's ingress.yaml

...
kind: Ingress
metadata:
  name: {{ $fullName }}
  labels:
    {{- include "ghost.labels" . | nindent 4 }}
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: {{ $maxSize }}
  {{- with .Values.ingress.annotations }}
    {{- toYaml . | nindent 4 }}
  {{- end }}
...

Based on this, I would say that I need to edit the annotations section of values.yaml to use traefik instead of default kubernetes.io/ingress.class: nginx

ingress:
  enabled: true
  annotations: 
    kubernetes.io/ingress.class: traefik  # what should I use?
  hosts:
    - host: some.domain.io
      paths:
        - /
  tls:
    - secretName: chart-example-tls
      hosts:
        - some.domain.io

However, no matter what I try in annotations, helm install keep getting

Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": service "ingress-nginx-controller-admission" not found

as if helm were looking for an Nginx ingress controler, no matter what. Is this a possible limitation of charts (like this one) or am I misconceiving how Helm charts work? How can I use the default traefik ingress on this chart's deployment?

Vit Vit · Accepted Answer · 2020-11-25T13:54:46

No, as per I know - you are not able to chose exact ingress controller for these chart. Both Nginx and Traefik use same ports, so you have to define proper ingress controller in advance.

From chart documentation i see that :

This chart has been tested to work with NGINX Ingress, cert-manager, bla bla

And the helm options you have are:

From all the above written I have a question|proposition: Dont you want to switch to NGINX ingress controller in K3s?

Getting your cluster ready First order of business is that you need to get your cluster have no other ingress controllers as this would conflict with your current ingress controllers. This is because NGINX and Traefik both listens to 80 and 443, which causes conflicts with each other, so make sure your K3s cluster only has one kind of the other.

For the best results, your K3s cluster must be installed with the --no-deploy-traefik argument, which will cause the K3s cluster to deploy only with the bare container orchestrator without the default HTTP backend, which is Traefik.

To make sure you don't have a Traefik controller installed, run kubectl get deployments -n kube-system to see if Traefik is gone.

If you have verified Traefik is not there, then you may proceed with the next part.

Installing NGINX There's two ways to install NGINX, either from a Helm chart or a direct kubectl apply -f

Using a Helm chart The NGINX ingress is available in the Helm Stable Charts, and if you have the stable registry already installed in your Helm CLI, simply run helm install stable/nginx-ingress.

You may also modify the installation using the following configurations stated here.

How to choose ingress controller in values.yaml

Context

The issue

2 Answers