Default Traefik ingress controller on k3s gives “Gateway Timeout”

Question

I have set up a simple two-node Kubernetes cluster using K3S. I have deployed a very simple web app, but when I try to access the web app, I just get a "Gateway Timeout". I've tried to keep the set up as simple as possible, but I can't see where I'm going wrong. I've outlined my entire setup below, from starting with two brand new Ubuntu 20.04 instances. Can anyone see where I'm going wrong?

This is my set up from start to finish:

On Master Node:

sudo ufw allow 80
sudo ufw allow 8080
sudo ufw allow 6443
sudo ufw allow 2379
sudo ufw allow 2380
sudo ufw allow 2379:10252/tcp
sudo ufw allow 30000:32767/tcp
export http_proxy=proxy.example.com:8082
export https_proxy=proxy.example.com:8082
curl -sfL https://get.k3s.io | sh -
cat /var/lib/rancher/k3s/server/node-token
sudo cat /var/lib/rancher/k3s/server/node-token
sudo cat /etc/rancher/k3s/k3s.yaml

On Agent:

sudo ufw allow 80
sudo ufw allow 8080
sudo ufw allow 6443
sudo ufw allow 2379
sudo ufw allow 2380
sudo ufw allow 2379:10252/tcp
sudo ufw allow 30000:32767/tcp
export http_proxy=proxy.example.com:8082
export https_proxy=proxy.example.com:8082
curl -sfL https://get.k3s.io | K3S_URL=https://vm1234.example.com:6443 K3S_TOKEN=K1060cf9217115ce1cb67d8450ea809b267ddc332b59c0c8ec6c6a30573f0b75eca::server:0b2be94c380be7bf4e16d94af36cac00 sh -
mkdir /etc/rancher/k3s/
sudo mkdir /etc/rancher/k3s/
sudo vim /etc/rancher/k3s/registries.yaml
sudo systemctl restart k3s-agent

On Local Workstation:

kubectl --kubeconfig k3s.yaml apply -f web-test-deployment.yaml
kubectl --kubeconfig k3s.yaml apply -f web-test-service.yaml 
kubectl --kubeconfig k3s.yaml apply -f web-test-ingress.yaml

List running pods:

$ kubectl --kubeconfig k3s.yaml get po
NAME                                   READY   STATUS    RESTARTS   AGE
web-test-deployment-5594bffd47-2gpd2   1/1     Running   0          4m57s

Inspect running pod:

$ kubectl --kubeconfig k3s.yaml describe pod web-test-deployment-5594bffd47-2gpd2
Name:         web-test-deployment-5594bffd47-2gpd2
Namespace:    default
Priority:     0
Node:         vm9876/10.192.110.200
Start Time:   Fri, 28 Aug 2020 12:07:01 +0100
Labels:       app=web-test
              pod-template-hash=5594bffd47
Annotations:  <none>
Status:       Running
IP:           10.42.1.3
IPs:
  IP:           10.42.1.3
Controlled By:  ReplicaSet/web-test-deployment-5594bffd47
Containers:
  web-test:
    Container ID:   containerd://c32d85da0642d3ccc00c61a5265280f9fcc11e8979d621690117878c89506440
    Image:          docker.example.com//web-test
    Image ID:       docker.example.com//web-test@sha256:cb568f5b6554284684815fc4ee17eb8cceb1aa90838a575fd3755b60bb7e44e7
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 28 Aug 2020 12:09:03 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wkzpx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wkzpx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wkzpx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                      Message
  ----     ------     ----                   ----                      -------
  Normal   Scheduled  <unknown>              default-scheduler         Successfully assigned default/web-test-deployment-5594bffd47-2gpd2 to vm9876
  Normal   Pulling    3m58s (x4 over 5m17s)  kubelet, vm9876  Pulling image "docker.example.com/web-test"
  Normal   Pulled     3m16s                  kubelet, vm9876  Successfully pulled image "docker.example.com/web-test"
  Normal   Created    3m16s                  kubelet, vm9876  Created container web-test
  Normal   Started    3m16s                  kubelet, vm9876  Started container web-test

Show stack:

$ kubectl --kubeconfig k3s.yaml get all
NAME                                       READY   STATUS    RESTARTS   AGE
pod/web-test-deployment-5594bffd47-2gpd2   1/1     Running   0          5m43s

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubernetes         ClusterIP   10.43.0.1       <none>        443/TCP    16m
service/web-test-service   ClusterIP   10.43.100.212   <none>        8080/TCP   5m39s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web-test-deployment   1/1     1            1           5m44s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/web-test-deployment-5594bffd47   1         1         1       5m45s

List Ingress:

$ kubectl --kubeconfig k3s.yaml get ing
NAME       CLASS    HOSTS   ADDRESS         PORTS   AGE
web-test   <none>   *       10.94.230.224   80      5m55s

Inspect Ingress:

$ kubectl --kubeconfig k3s.yaml describe ing web-test
Name:             web-test
Namespace:        default
Address:          10.94.230.224
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /   web-test-service:8080 (10.42.1.3:8080)
Annotations:  kubernetes.io/ingress.class: traefik
Events:       <none>

Inspect Service:

kubectl --kubeconfig k3s.yaml describe svc web-test-service
Name:              web-test-service
Namespace:         default
Labels:            app=web-test
Annotations:       Selector:  app=web-test
Type:              ClusterIP
IP:                10.43.100.212
Port:              <unset>  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.42.1.3:8080
Session Affinity:  None
Events:            <none>

$ curl http://10.94.230.224/web-test-service/

Gateway Timeout

These are my deployment manifests:

web-test-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web-test
  name: web-test-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-test
  strategy: {}
  template:
    metadata:
      labels:
        app: web-test
    spec:
      containers:
      - image: docker.example.com/web-test
        imagePullPolicy: Always
        name: web-test
        ports:
        - containerPort: 8080
      restartPolicy: Always
      volumes: null

web-test-service.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: web-test
  name: web-test-service
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: web-test

web-test--ingress.yaml:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: web-test
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: web-test-service
          servicePort: 8080

Note: I've also tried a similar setup using Ambassador, but I'm getting similar results :-(

Can you add exact curl command(s) you're doing to ping and responses? — taleodor
curl http://10.94.230.224/web-test-service/ -- I've rewritten the question to give more detail so you can follow from start to finish. From reading the docs, the above should work for anyone that has a working docker image that will expose a http endpoint on port 8080. BTW: I did check the logs and the docker container is running properly. — simbro
What you have looks correct, don't see anything obviously wrong. Things you may want to check: try curl'ing on service IP from within your node (should work), try curl'ing on localhost from within the node (should work). Basically, k3s should expose ingress on node interface, so you should be able to hit it on node's ip or localhost. Also when you're doing curl to ingres IP as above, make sure you're doing it from a k3s node itself (it's not supposed to work outside). — taleodor

DarthHTTP DarthHTTP · Accepted Answer · 2021-02-08T11:20:56

The annotation on the Ingress is missing description of the entrypoint and the host:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
    name: web-test
    annotations:
        kubernetes.io/ingress.class: "traefik"
        traefik.ingress.kubernetes.io/router.entrypoints: http
    spec:
      rules:
        - host: webtest.example.com
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: web-test-service
                    port:
                      number: 80

Default Traefik ingress controller on k3s gives “Gateway Timeout”

1 Answers