I have a VM on which k8s v1.16.2 is deployed and on top of it prometheus and grafana services running. On grafana dashboard i am able to see all containers and pods performance metrics but for system services only docker and kubelet, Why its not showing other services metrics which are running directly on machine.
Is there any configurations by default kubelet sets to cadvisor which restricts collecting system service metrics..? because with kubelet version v1.13.1 i was able to see system services metrics.
How can i get all service metrics which runs under system.slice through cadvisor of kubelet..?
Verified through :
- Hitting directly Prometheus endpoints.
- Tried directly cadvisor endpoint(k8s exposed endpoint) : /api/v1/nodes/HOSTNAME/proxy/metrics/cadvisor
on both endpoints results i am not seeing any system service metrics except docker and kubelet
Version:
- Docker: 18.09.6,
- Kubelet: v1.16.2,
- CGroup Driver: systemd
Below is Prometheus Configuration:
metric_relabel_configs:
- source_labels: [id]
separator: ;
regex: ^/machine\.slice/machine-rkt\\x2d([^\\]+)\\.+/([^/]+)\.service$
target_label: rkt_container_name
replacement: ${2}-${1}
action: replace
- source_labels: [id]
separator: ;
regex: ^/system\.slice/(.+)\.service$
target_label: systemd_service_name
replacement: ${1}
action: replace
apiVersionchanges. It stopped working just after you upgraded your Kubernetes and docker version? – PjoterS