Setting up Kubernetes Autoscaling with Custom Metrics

Question

I am trying to set up autoscaling on a Kubernetes 1.2.3 (beta) cluster based on custom metrics. (I already tried CPU-based autoscaling on the cluster, and it worked fine.)

I tried to follow their custom metrics proposal, but I'm having problems in creating the necessary set-up.

This is what I have done so far:

Added a custom metrics annotation to the pod spec being deployed (similar to the configuration provided in their proposal):

apiVersion: v1
kind: ReplicationController
metadata:
  name: metrix
  namespace: "default"
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: metrix
      annotations:
        metrics.alpha.kubernetes.io/custom-endpoints: >
          [
            {
              "api": "prometheus",
              "path": "/status",
              "port": "9090",
              "names": ["test1"]
            },
            {
              "api": "prometheus",
              "path": "/metrics",
              "port": "9090"
              "names": ["test2"]
            }
          ]
    spec:
      containers:
      - name: metrix
        image: janaka/prometheus-ep:v1
        resources:
          requests:
            cpu: 400m

Created a Docker container tagged janaka/prometheus-ep:v1 (local) running a Prometheus-compatible server on port 9090, with /status and /metrics endpoints
Enabled custom metrics on the kubelet by appending --enable-custom-metrics=true to KUBELET_OPTS at /etc/default/kubelet (based on the kubelet CLI reference) and restarted the kubelet

All pods (in default and kube-system namespaces) are running, and the heapster pod log doesn't contain any 'anomalous' outputs either (except for a small glitch at startup, due to temporary unavailability of InfluxDB):

$ kubesys logs -f heapster-daftr

I0427 05:07:45.807277       1 heapster.go:60] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0427 05:07:45.807359       1 heapster.go:61] Heapster version 1.1.0-beta1
I0427 05:07:45.807638       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version "v1"
I0427 05:07:45.807661       1 configs.go:61] Using kubelet port 10255
E0427 05:08:15.847319       1 influxdb.go:185] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp xxx.xxx.xxx.xxx:8086: i/o timeout, will retry on use
I0427 05:08:15.847376       1 influxdb.go:199] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0427 05:08:15.847412       1 heapster.go:87] Starting with InfluxDB Sink
I0427 05:08:15.847427       1 heapster.go:87] Starting with Metric Sink
I0427 05:08:15.877349       1 heapster.go:166] Starting heapster on port 8082
I0427 05:08:35.000342       1 manager.go:79] Scraping metrics start: 2016-04-27 05:08:00 +0000 UTC, end: 2016-04-27 05:08:30 +0000 UTC
I0427 05:08:35.035800       1 manager.go:152] ScrapeMetrics: time: 35.209696ms size: 24
I0427 05:08:35.044674       1 influxdb.go:177] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0427 05:09:05.000441       1 manager.go:79] Scraping metrics start: 2016-04-27 05:08:30 +0000 UTC, end: 2016-04-27 05:09:00 +0000 UTC
I0427 05:09:06.682941       1 manager.go:152] ScrapeMetrics: time: 1.682157776s size: 24
I0427 06:43:38.767146       1 manager.go:79] Scraping metrics start: 2016-04-27 05:09:00 +0000 UTC, end: 2016-04-27 05:09:30 +0000 UTC
I0427 06:43:38.810243       1 manager.go:152] ScrapeMetrics: time: 42.940682ms size: 1
I0427 06:44:05.012989       1 manager.go:79] Scraping metrics start: 2016-04-27 06:43:30 +0000 UTC, end: 2016-04-27 06:44:00 +0000 UTC
I0427 06:44:05.063583       1 manager.go:152] ScrapeMetrics: time: 50.368106ms size: 24
I0427 06:44:35.002038       1 manager.go:79] Scraping metrics start: 2016-04-27 06:44:00 +0000 UTC, end: 2016-04-27 06:44:30 +0000 UTC

However, the custom endpoints are not being scraped. (I verified it by adding stderr logs for startup and endpoint handlers of my server; only the server initialization logs are displayed on kubectl logs of the pod.)

As I'm a newbie to Kubernetes, any help is appreciated.

(From what I understood from the proposal as well as this issue, we don't have to run a separate Prometheus collector in the cluster as cAdvisor should already pull data from the endpoints defined in the pod spec. Is this true, or do I need a separate Prometheus collector as well?)

Jerzy Szczepkowski Jerzy Szczepkowski · Accepted Answer · 2016-05-04T12:25:00

The custom metrics proposal is out of date.

Please refer to the user guide, which is currently under review.

Setting up Kubernetes Autoscaling with Custom Metrics

1 Answers