How to calculate cpu usage of pod in kube state metrics?

Question

I monitor the eks cluster using both the kubernetes api and the kube state metrics remotely using prometheus. in Kubernetes api we have a metrics container_cpu_usage_seconds_total which gives the cpu usage of the pod. is there a similar metrics in kube-state-metrics which can give the cpu usage. Actually I'm trying to get the cluster cpu usage which is totally different from kubernetes api and kube-state-metrics following are the calculations.

kube-state-metrics:

sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_allocatable_cpu_cores) * 100 - This gives 101%

whereas the kube-state-metrics gives 12% which looks accurate to me.

kubernetes-api:

sum (rate (container_cpu_usage_seconds_total{id="/",kubernetes_io_hostname=~"^$Node$", job=~"$job$"}[5m])) / sum (machine_cpu_cores{kubernetes_io_hostname=~"^$Node$", job=~"$job$"}) * 100

I don't think there's any metric in kube-state-metric which gives cpu usage compared to kubernetes-api

Thanks in advance.

Answer 1

No, there is no (one) specific metric for cpu usage per container in kube_state_metric.

The value you got: sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_allocatable_cpu_cores) * 100 = 101 may be wrong because metrics like kube_node_status_allocatable_cpu_cores and kube_pod_container_resource_requests_cpu_cores are marked as DEPRECATED.

At the same time, take notice that there is metric like kube_pod_container_resource_limits_cpu_cores. Your container could have set resource limit, that's why probably your result exceed 100 %. If you have limit set per container check if resource limit is lower that resource request and then your calculation should looks like: [sum(kube_pod_container_resource_requests_cpu_cores) - sum(kube_pod_container_resource_limits_cpu_cores)]/ sum(kube_node_status_allocatable_cpu_cores) * 100.

Take a look for every resource metrics in kube_state_metrics for container and nodes: node_metrics, pod_container_metrics.