Prometheus is not compatible with Kubernetes v1.16

0
votes

I installed the stable/prometheus helm chart with some minor changes proposed at helm/charts#17268 to make it compatible with Kubernetes v1.16

After installation, none of the Kubernetes grafana dashboards show correct values. I am using 8769 (https://grafana.com/grafana/dashboards/8769) dashboard which provides many information on cpu, memory, network, etc. This dashboard is working properly on older k8s versions but on v1.16 it shows no results. I also randomly tried some other dashboards (8588, 6879, 10551) but they either just show the requested resource for each pod and not the live usage or showing nothing.

What these dashboards do is they send a promql query to prometheus and get the results. For example this is the promql query for cpu usage from 8769 dashboard:

sum (rate (container_cpu_usage_seconds_total{id!="/",namespace=~"$Namespace",pod_name=~"^$Deployment.*$"}[1m])) by (pod_name)

I don't know if I have to change the promql or the problem is somewhere else.

2
kubernetesgrafanaprometheuspromql
Can you check whether prometheus target source status is Up or not? You can also check from Prometheus UI by running simple query like, container_cpu_usage_seconds_total. – Kamol Hasan
Everything is up and running, the problem is the new changes that were introduced in k8s v1.16. – AVarf

2 Answers

7
votes

Kubernetes 1.16 removes the labels pod_name and container_name from cAdvisor metrics, duplicates of pod and container.

You need change pod_name -> pod, container_name -> container in Grafana dashboards JSON models.

0
votes

Try the installation this way, as the new CRDs had some issue, so I used old CRDs- 

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/servicemonitor.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.32/example/prometheus-operator-crd/podmonitor.crd.yaml

helm install --name prometheus --namespace monitoring  stable/prometheus-operator --set prometheusOperator.createCustomResource=false

Make sure that CRD's don't exist priory, you can delete them via

kubectl delete crd --all