Kubernetes Dashboard not accessible when providing path in ingress

Question

I have deployed minikube on Windows VM and the minikube VM is created on Virtualbox with the host-only IP.

I have deployed the Kubernetes dashboard with NodePort IP so I can access it from outside the cluster. The svc is as follows:

PS C:\Users\XXX\Desktop\ingress> kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.111.167.61   <none>        8000/TCP        5d20h
kubernetes-dashboard        NodePort    10.111.220.57   <none>        443:30613/TCP   5d20h

With the help of the minikube ingress addon, I installed the Ingress controller which is of Nginx. Its svc details are as follows:

PS C:\Users\XXX\Desktop\ingress> kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.98.29.41   <none>        80:32628/TCP,443:31194/TCP   5d20h
ingress-nginx-controller-admission   ClusterIP   10.96.35.36   <none>        443/TCP                      5d20h

Then I have created an Ingress Rule for my dashboard application as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    ingress.kubernetes.io/configuration-snippet: |
      rewrite ^(/dashboard)$ $1/ permanent;
spec:
  rules:
  - host: k8s.dashboard.com
    http:
      paths:
      - path: /dashboard
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

But now when I am trying to access the dashboard with the following URL https://k8s.dashboard.com/dashboard then I am facing the error of 404 Not Found. I also tried multiple URL to access the dashboard such as :

https://k8s.dashboard.com:30613/dashboard
http://k8s.dashboard.com:30613/dashboard
https://k8s.dashboard.com/dashboard

But this URL is working for me: https://k8s.dashboard.com:30613 I have added the minikube IP to hosts files in the Windows machine. Ingress rule describe the output is as follows:

PS C:\Users\XXX\Desktop\ingress> kubectl describe ingress -n kubernetes-dashboard
Name:             dashboard-ingress
Namespace:        kubernetes-dashboard
Address:          192.168.56.100
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  k8s.dashboard.com
                     /dashboard   kubernetes-dashboard:443 (172.17.0.4:8443)
Annotations:         ingress.kubernetes.io/configuration-snippet: rewrite ^(/dashboard)$ $1/ permanent;
                     kubernetes.io/ingress.class: nginx
                     nginx.ingress.kubernetes.io/add-base-url: true
                     nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                     nginx.ingress.kubernetes.io/force-ssl-redirect: false
                     nginx.ingress.kubernetes.io/rewrite-target: /
                     nginx.ingress.kubernetes.io/secure-backends: true
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    26m (x16 over 5d20h)  nginx-ingress-controller  Scheduled for sync

Any help regarding this really helps. Thanks

EDITED My ingress controller logs are as follows:

192.168.56.1 - - [16/Jun/2021:06:57:00 +0000] "GET /dashboard HTTP/2.0" 200 746 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 418 0.019 [kubernetes-dashboard-kubernetes-dashboard-443] [] 172.17.0.4:8443 746 0.018 200 1a2793052f70031c6c9fa59b0d4374d1
192.168.56.1 - - [16/Jun/2021:06:57:00 +0000] "GET /styles.aa1f928b22a88c391404.css HTTP/2.0" 404 548 "https://k8s.dashboard.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 101 0.002 [upstream-default-backend] [] 127.0.0.1:8181 548 0.002 404 1974258442f8b4c46d8badd1dda3e3f5
192.168.56.1 - - [16/Jun/2021:06:57:00 +0000] "GET /runtime.2a456dd93bf6c4890676.js HTTP/2.0" 404 548 "https://k8s.dashboard.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 49 0.008 [upstream-default-backend] [] 127.0.0.1:8181 548 0.007 404 96c17c52e6337f29dd8b2b2b68b088ac
192.168.56.1 - - [16/Jun/2021:06:57:00 +0000] "GET /polyfills.f4f05ad675be9638106e.js HTTP/2.0" 404 548 "https://k8s.dashboard.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 40 0.008 [upstream-default-backend] [] 127.0.0.1:8181 548 0.007 404 096ae29cb168523aa9191f27a967e47a
192.168.56.1 - - [16/Jun/2021:06:57:00 +0000] "GET /scripts.128068f897fc721c4673.js HTTP/2.0" 404 548 "https://k8s.dashboard.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 38 0.008 [upstream-default-backend] [] 127.0.0.1:8181 548 0.007 404 728f73f75276167b387dc87a69b65a72
192.168.56.1 - - [16/Jun/2021:06:57:00 +0000] "GET /en.main.09bf52db2dbc808e7279.js HTTP/2.0" 404 548 "https://k8s.dashboard.com/dashboard" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" 38 0.014 [upstream-default-backend] [] 127.0.0.1:8181 548 0.014 404 b11e5ae324a828508d488816306399c2

and this is dashboard logs

172.17.0.1 - - [16/Jun/2021:06:59:46 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.20"
172.17.0.1 - - [16/Jun/2021:06:59:56 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.20"
172.17.0.1 - - [16/Jun/2021:07:00:00 +0000] "GET /healthz HTTP/1.1" 200 13 "" "dashboard/v2.2.0"
172.17.0.1 - - [16/Jun/2021:07:00:06 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.20"
172.17.0.1 - - [16/Jun/2021:07:00:16 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.20"
172.17.0.1 - - [16/Jun/2021:07:00:26 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.20"
172.17.0.1 - - [16/Jun/2021:07:00:30 +0000] "GET /healthz HTTP/1.1" 200 13 "" "dashboard/v2.2.0"
172.17.0.1 - - [16/Jun/2021:07:00:36 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.20"
{"level":"error","msg":"Error scraping node metrics: the server could not find the requested resource (get nodes.metrics.k8s.io)","time":"2021-06-16T07:00:41Z"}

Can you share the nginx ingress controller logs showing the 404 error? Furthermore, do you see anything in the dashboard logs when you get the 404 in the browser? — whites11
Seems a limitation of the dashboard itself. The frontend is requesting stylesheets and JS files without the /dashboard prefix and thus nginx 404s. Try this github.com/kubernetes/dashboard/issues/… — whites11
It WORKED! But I am not sure what does /dashboard(/|$)(.*) means. And why it is not working normally like other applications we deploy on k8s. Anyway Thanks a lot! — Sakar Mehra

whites11 whites11 · Accepted Answer · 2021-06-16T08:45:26

According to this issue this is a limitation/bug of the kubernetes dashboard.

They suggest using this config as a workaround:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  labels:
    app.kubernetes.io/name: kubernetes-dashboard  
  annotations:
    kubernetes.io/ingress.class: nginx
    # Add https backend protocol support for ingress-nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Accept-Encoding "";
      sub_filter '<base href="/">' '<base href="/dashboard/">';
      sub_filter_once on;
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: my.example.com
      http:
        paths:
          - path: /dashboard(/|$)(.*)
            backend:
              serviceName: kubernetes-dashboard
              servicePort: 443

Kubernetes Dashboard not accessible when providing path in ingress

1 Answers