Kubernetes Prometheus metric for HPA (horizontal pod autoscaler) `currentCPUUtilizationPercentage`?

Question

On a Kubernetes cluster with CoreOS Prometheus Operator scraping all standard cluster metrics, what Prometheus metric would show me the currentCPUUtilizationPercentage value for a simple HPA (horizontal pod autoscaler)?

If I setup a simple hpa like:

$ kubectl autoscale deployment php-apache --cpu-percent=30 --min=3 --max=10

And then if I do kubectl get hpa php-apache -o yaml I see something like:

spec:
  maxReplicas: 10
  minReplicas: 3
  targetCPUUtilizationPercentage: 30
  ...
status:
  currentCPUUtilizationPercentage: 28
  currentReplicas: 9
  desiredReplicas: 9
  ...

I want to see that currentCPUUtilizationPercentage in Prometheus. I've done a bunch of Prometheus queries to look for this.

I've searched all metrics tagged {hpa="php-apache"} and I see many hpa metrics, but not the metric I'm looking or. I can see kube_hpa_spec_target_metric set to 30 and I can see current status metrics like kube_hpa_status_condition but not the current cpu metric value that I want to see.

I've searched all metrics tagged {metric_name="cpu"} and only see kube_hpa_spec_target_metric

I've searched all container and pod related metrics tagged {container="my-container-name"} and {pod=~"my-pod-prefix.*"} and I see several cpu related metrics like container_cpu_usage_seconds_total abnd container_spec_cpu_quota but nothing similar to or nothing that seems to be able to be used in calculating the currentCPUUtilizationPercentage value that I'm looking for?

FYI, this is on Kubernetes 1.17.x and using a recent version of CoreOS Prometheus Operator.

Vlad Ulshin Vlad Ulshin · Accepted Answer · 2020-08-30T01:18:37

If I remember correctly currentCPUUtilizationPercentage is a k8 internal metrics for HPA and not exposed directly as a metrics you can scrape with Prometheus. see https://github.com/kubernetes/community/blob/master/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#autoscaling-algorithm

you probably could scrape component of currentCPUUtilizationPercentage metrics and create custom metrics to see it in Prometheus.

Kubernetes Prometheus metric for HPA (horizontal pod autoscaler) `currentCPUUtilizationPercentage`?

1 Answers