kubernetes dashboard reachable with curl but not chrome/firefox via kubectl proxy

Question

I have two clusters that are very different (one kubernetes 1.13 with dashboard 1.0 and created with kops in aws; the other uses kubernetes 1.14 with dashboard 2.0 and created with EKS) same issue for both, and I use kubectl 1.17 to interface with both. Once I start kubectl proxy, I can reach the dashboard I just installed via curl. Eg with the dashboard 2.0 in newer EKS cluster:

In one terminal:

$ kubectl proxy 
Starting to serve on 127.0.0.1:8001

In another terminal

$ curl http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
<!--
Copyright 2017 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<!doctype html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <title>Kubernetes Dashboard</title>
  <link rel="icon" type="image/png" href="assets/images/kubernetes-logo.png"/>
  <meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="styles.d8a1833bf9631b49f8ae.css"></head>

<body>
  <kd-root></kd-root>
<script src="runtime.a3c2fb5c390e8fb10577.js" defer=""></script><script src="polyfills-es5.ddf623b9f96429e29863.js" nomodule="" defer=""></script><script src="polyfills.24a7a4821c30c3b30717.js" defer=""></script><script src="scripts.391d299173602e261418.js" defer=""></script><script src="main.a0d83b15387cfc420c65.js" defer=""></script></body>

</html>

Clearly the dashboard service is reachable and responding to the request. The html is a bit different for the other cluster/dashboard combo but still no error.

However the exact same URL from chrome or firefox (running on the same host of course) gives me an error:

This site can’t be reached
127.0.0.1 refused to connect.
Try:
Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED

The dashboard 2.0 itself seems happy:

$ kubectl get all -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-76679bc5b9-k7qjp   1/1     Running   0          136m
pod/kubernetes-dashboard-565688d4c4-dtw5w        1/1     Running   0          136m

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/dashboard-metrics-scraper   ClusterIP   172.20.42.193    <none>        8000/TCP   142m
service/kubernetes-dashboard        ClusterIP   172.20.232.104   <none>        443/TCP    142m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dashboard-metrics-scraper   1/1     1            1           142m
deployment.apps/kubernetes-dashboard        1/1     1            1           142m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/dashboard-metrics-scraper-6c554969c6   0         0         0       137m
replicaset.apps/dashboard-metrics-scraper-76679bc5b9   1         1         1       142m
replicaset.apps/kubernetes-dashboard-565688d4c4        1         1         1       142m
replicaset.apps/kubernetes-dashboard-56c5f95c6b        0         0         0       137m

Any ideas what is wrong? How is it possible that it works with curl and not a web browser?

Updated info:

I checked ifconfig:

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ...    
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        ...

With the following kubectl proxy command, I can access dashboard in browser too:

In one terminal:

kubectl proxy --address='172.17.0.2' --accept-hosts='.*'

Then chrome browser to http://172.17.0.2:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ shows the login screen.

Both flags are necessary otherwise neither curl nor browser will work (the response is Forbidden if I don't use the --accept-hosts -- although that response is from the service so it is at least better than when using loopback).

Replacing by 127.0.0.1 by localhost does not help. I can reach the api server only if I used the full proxy command and http://172.17.0.2:8001/api.

Does anyone know why chrome will not handle 127.0.0.1, whereas curl does, and why this accept-hosts is necessary when curling 172.12 IP but not when using 127 IP?

Does it work if you use localhost instead of 12.0.0.1 in the browser?are you able to access anything else from browser via kubectl proxy?for example localhost:8001/api — Arghya Sadhu
@arghya Added info in the post, but basically no to both questions. — Oliver

Oliver Oliver · Accepted Answer · 2020-05-27T15:57:02

Well this is embarrassing but I figure I might as well leave this question and post an answer just in case someone else forgets the obvious:

I was shelled into a docker container running on my host when running the curl and kubectl proxy commands (I forgot! as it runs non-stop). The container shares the host's network which is why the 172.17 worked from the browser, but not the loopback.

If you setup the container to port-forward say port 8080 to 8080 (docker ... -p 8080:8080 ...) then the following proxy command (from inside that container) also works:

$ kubectl proxy --port 8080 --address='0.0.0.0'

ie browsing to http://localhost:8080/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login on host works.

Sorry for the mistake!

kubernetes dashboard reachable with curl but not chrome/firefox via kubectl proxy

1 Answers