Kubernetes horizontal pod autoscaling not working

Question

kubectl version Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"} Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"}

Error log on controller manager: Failed to reconcile hello-world: failed to compute desired number of replicas based on CPU utilization for Deployment/default/hello-world: failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/3 of pods

I can talk directly to Heapster and get my pod metrics: curl http://$MY_HEAPSTER_SERVICE_IP:8082/api/v1/model/namespaces/default/pods//hello-world-819237062-j0ubt/metrics/cpu-usage

{
  "metrics": [
   {
    "timestamp": "2016-04-05T16:49:00Z",
    "value": 0
   },
   {
    "timestamp": "2016-04-05T16:48:00Z",
    "value": 0
   }
  ],
  "latestTimestamp": "2016-04-05T16:49:00Z"
 }

After looking at the controller manager code, I'm able to figure out the reason for this failure. Controller manager is hardcoded to lookup heapster service with the name "heapster" and in "kube-system" namespace. My service was not named heapster and so controller manager was unable to communicate with it. — Dhawal

Filip Filip · Accepted Answer · 2016-04-06T21:30:21

As explained in the comment above the Heapster was running in a non-default location, which horizontal pod autoscaler controller did not expect.

I've opened an issue to make it configurable.

Kubernetes horizontal pod autoscaling not working

1 Answers