Heapster fails to get container stats from Kubelet on Kubernetes cluster

2
votes

I've set up a Kubernetes cluster on Ubuntu (trusty) based on the Running Kubernetes Locally via Docker guide, deployed a DNS and run Heapster with an InfluxDB backend and a Grafana UI.

Everything seems to run smoothly except for Grafana, which doesn't show any graphs but the message No datapoints in its diagrams: Screenshot

After checking the Docker container logs I found out that Heapster is is unable to access the kubelet API (?) and therefore no metrics are persisted into InfluxDB:

user@host:~$ docker logs e490a3ac10a8
I0701 07:07:30.829745       1 heapster.go:65] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0701 07:07:30.830082       1 heapster.go:66] Heapster version 1.2.0-beta.0
I0701 07:07:30.830809       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0701 07:07:30.831284       1 configs.go:61] Using kubelet port 10255
E0701 07:09:38.196674       1 influxdb.go:209] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out, will retry on use
I0701 07:09:38.196919       1 influxdb.go:223] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0701 07:09:38.197048       1 heapster.go:92] Starting with InfluxDB Sink
I0701 07:09:38.197154       1 heapster.go:92] Starting with Metric Sink
I0701 07:09:38.228046       1 heapster.go:171] Starting heapster on port 8082
I0701 07:10:05.000370       1 manager.go:79] Scraping metrics start: 2016-07-01 07:09:00 +0000 UTC, end: 2016-07-01 07:10:00 +0000 UTC
E0701 07:10:05.008785       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:10:05.009119       1 manager.go:152] ScrapeMetrics: time: 8.013178ms size: 0
I0701 07:11:05.001185       1 manager.go:79] Scraping metrics start: 2016-07-01 07:10:00 +0000 UTC, end: 2016-07-01 07:11:00 +0000 UTC
E0701 07:11:05.007130       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:11:05.007686       1 manager.go:152] ScrapeMetrics: time: 5.945236ms size: 0
W0701 07:11:25.010298       1 manager.go:119] Failed to push data to sink: InfluxDB Sink
I0701 07:12:05.000420       1 manager.go:79] Scraping metrics start: 2016-07-01 07:11:00 +0000 UTC, end: 2016-07-01 07:12:00 +0000 UTC
E0701 07:12:05.002413       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:12:05.002467       1 manager.go:152] ScrapeMetrics: time: 1.93825ms size: 0
E0701 07:12:12.309151       1 influxdb.go:150] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out
I0701 07:12:12.351348       1 influxdb.go:201] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0701 07:13:05.001052       1 manager.go:79] Scraping metrics start: 2016-07-01 07:12:00 +0000 UTC, end: 2016-07-01 07:13:00 +0000 UTC
E0701 07:13:05.015947       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
...

I found a few issues on GitHub describing similar problems that made me understand that Heapster doesn't access the kubelet (via the node's loopback) but itself (via the container's loopback) instead. However, I fail to reproduce their solutions:

github.com/kubernetes/heapster/issues/1183

You should either use host networking for Heapster pod or configure your cluster in a way that the node has a regular name not 127.0.0.1. The current problem is that node name is resolved to Heapster localhost. Please reopen in case of more problems.

-@piosz

  • How do I enable "host networking" for my Heapster pod?
  • How do I configure my cluster/node to use a regular name not 127.0.0.1?

github.com/kubernetes/heapster/issues/744

Fixed by using better options in hyperkube, thanks for the help!

-@ddispaltro

  • Is there a way to solve this issue by adding/modifying kubelet's option flags in docker run?
    I tried setting--hostname-override=<host's eth0 IP> and --address=127.0.0.1 (as suggested in the last answer of this GitHub issue) but Heapster's container log then states:

    I0701 08:23:05.000566 1 manager.go:79] Scraping metrics start: 2016-07-01 08:22:00 +0000 UTC, end: 2016-07-01 08:23:00 +0000 UTC E0701 08:23:05.000962 1 kubelet.go:279] Node 127.0.0.1 is not ready E0701 08:23:05.003018 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://<host's eth0 IP>:10255/stats/container/": Post http://<host's eth0 IP>/stats/container/: dial tcp <host's eth0 IP>:10255: getsockopt: connection refused

Namespace issue

Could this problem be caused by the fact that I'm running Kubernetes API in default namespace and Heapster in kube-system?

user@host:~$ kubectl get --all-namespaces pods
NAMESPACE     NAME                     READY     STATUS    RESTARTS   AGE
default       k8s-etcd-127.0.0.1       1/1       Running   0          18h
default       k8s-master-127.0.0.1     4/4       Running   1          18h
default       k8s-proxy-127.0.0.1      1/1       Running   0          18h
kube-system   heapster-lizks           1/1       Running   0          18h
kube-system   influxdb-grafana-e0pk2   2/2       Running   0          18h
kube-system   kube-dns-v10-4vjhm       4/4       Running   0          18h

OS: Ubuntu 14.04.4 LTS (trusty) | Kubernetes: v1.2.5 | Docker: v1.11.2

2
dockerkubernetesheapster
The "kubernetes" service appears in all namespaces, but points to the same thing. I don't think that is your problem. The "locally" guide is sort of a toy and we don't run automated testing against it, so it is quite possible that this just doesn't work on locally. – Eric Tune
@EricTune thanks a lot for your answer, at least I can now ignore the supposed namespace issue. – IggyBlob

2 Answers

1
votes

Heapster has got the list of nodes from Kubernetes and is now trying to pull stats from the kublete process on each node (which has a built in cAdvisor collecting stats on the node). In this case there's only one node and it's known by 127.0.0.1 to kubernetes. And there's the problem. The Heapster container is trying to reach the node at 127.0.0.1 which is itself and of course finding no kublete process to interrogate within the Heapster container.

Two things need to happen to resolve this issue.

  1. We need to reference the kublete worker node (our host machine running kubernetes) by something else other than the loopback network address of 127.0.0.1
  2. The kublete process needs to accept traffic from the new network interface/address

Assuming you are using the local installation guide and starting kubernetes off with

hack/local-up-cluster.sh

To change the hostname by which the kublete is referenced is pretty simple. You can take more elaborate approaches but setting this to your eth0 ip worked fine for me (ifconfig eth0). The downside is that you need a eth0 interface and this is subject to DHCP so your mileage may vary as to how convenient this is. 

export HOSTNAME_OVERRIDE=10.0.2.15

To get the kublete process to accept traffic from any network interface is just as simple. 

export KUBELET_HOST=0.0.0.0
1
votes

Provide the below argument to your heapster configuration to resolve the issue.

--source=kubernetes:https://kubernetes.default:443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true