kubernetes autoscale fails to get metrics (heapster already installed in kube-system namespace)

Question

I created a mini cluster with vagrant and centos 7. I managed to install kube-dns and heapster but when i try to test an autoscale with the php-apache example it doesnt work.

failed to get CPU consumption and request: failed to unmarshall heapster response: invalid character 'E' looking for beginning of value

That's odd because I can see the metrics with grafana and the limits. My kube-dns and my heapster are in kube-system namespace so it should work.

I have kubernetes 1.2, if someone can help that will be awesome.

Here are the logs of heapster :

[vagrant@localhost ~]$ kubectl logs --namespace=kube-system heapster-xzy31
I0625 11:08:50.041788       1 heapster.go:65] /heapster --source=kubernetes:http://192.168.50.130:8080?inClusterConfig=false&useServiceAccount=true&auth= --sink=influxdb:http://monitoring-influxdb:8086
I0625 11:08:50.042310       1 heapster.go:66] Heapster version 1.1.0
I0625 11:08:50.090679       1 configs.go:60] Using Kubernetes client with master "http://192.168.50.130:8080" and version v1
I0625 11:08:50.090705       1 configs.go:61] Using kubelet port 10255
E0625 11:09:00.097603       1 influxdb.go:209] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp: lookup monitoring-influxdb on 10.254.0.10:53: read udp 172.17.39.2:38757->10.254.0.10:53: read: connection refused, will retry on use
I0625 11:09:00.097624       1 influxdb.go:223] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0625 11:09:00.097638       1 heapster.go:92] Starting with InfluxDB Sink
I0625 11:09:00.097641       1 heapster.go:92] Starting with Metric Sink
I0625 11:09:00.103486       1 heapster.go:171] Starting heapster on port 8082
I0625 11:10:05.003399       1 manager.go:79] Scraping metrics start: 2016-06-25 11:09:00 +0000 UTC, end: 2016-06-25 11:10:00 +0000 UTC
E0625 11:10:05.003479       1 kubelet.go:279] Node 192.168.50.131 is not ready
I0625 11:10:05.051081       1 manager.go:152] ScrapeMetrics: time: 47.581507ms size: 70
I0625 11:10:05.060501       1 influxdb.go:201] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0625 11:11:05.001120       1 manager.go:79] Scraping metrics start: 2016-06-25 11:10:00 +0000 UTC, end: 2016-06-25 11:11:00 +0000 UTC
I0625 11:11:05.091844       1 manager.go:152] ScrapeMetrics: time: 90.657932ms size: 132

What version of heapster are you running? Can you also paste the output of the heaspter logs (kubectl logs --namespace <namespace> <heapster_pod_name> -c <container_name>)? — Yu-Ju Hong
thanks for your help I edited the post to add the logs of heapster, I use heapster the logs show that it is version 1.1.0 — beorn

beorn beorn · Accepted Answer · 2016-06-26T13:25:36

Ok I found the solution, my cluster was flawed. I had to install flannel on the master too. With the option --iface=eth1 because of vagrant.

I followed this guide http://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services but they didnt say to install flannel on the master.

Now everything is working.

kubernetes autoscale fails to get metrics (heapster already installed in kube-system namespace)

1 Answers