HPA not able to fetch metrics from Prometheus in Kubernetes

0
votes

I have a two node Kubernetes cluster i.e one master node and two worker nodes. For monitoring purpose, I have deployed Prometheus and Grafana. Now, I want to autoscale pods based on CPU usage. But even after configuring Grafana and Prometheus, I am getting the following error ---

Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 17 Jun 2019 12:33:01 +0530
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  112s (x12408 over 2d4h)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Can anybody let me know why Kubernetes is not fetching metrics from Prometheus ?

3
kubernetesgrafanaprometheus

3 Answers

1
votes

Kubernetes retrieves metrics from either the metrics.k8s.io API (normally implemented by the metrics-server which can be seperatly installed) or the custom.metrics.k8s.io API (which can be any type of metric and is normally provided by third parties). To use prometheus in HPA for kubernetes the Prometheus Adapter for the custom metrics API needs to be installed.

A walkthrough for the setup can be found here.

1
votes

heapster is now depracted : https://github.com/kubernetes-retired/heapster

To enable auto-scaling on your cluster you can use HPA(horizontal pod auto-scaler) and you can also install metrics server to check all metrics.

To install metrics server on kubernetes you can follow this guide also :

amazon : https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html
https://github.com/kubernetes-incubator/metrics-server
https://medium.com/@cagri.ersen/kubernetes-metrics-server-installation-d93380de008
0
votes

You don't need custom metrics to use HPA for auto-scaling pods based on their CPU usage.

As @Blokje5 mentioned earlier, you just need to install 'kube-state-metrics'.

The most convenient way to do it is with a dedicated helm chart (kube-state-metrics).

Hint: use override parameters with 'helm install' to create ServiceMonitor object for 'kube-state-metrics' Pod, to allow Prometheus to discover a new target for metrics scraping, e.g.:

helm install stable/kube-state-metrics --set prometheus.monitor.enabled=true

Remark: Pay attention to the 'serviceMonitorSelector' defined in your existing Prometheus resource object/configuration, so that it matches the ServiceMonitor definition for 'kube-state-metrics'. This is to make available Pods' metrics in Prometheus console.