cAdvisor does not show all container's uptime (Prometheus+cAdvisor+Grafana)

0
votes

Environment:

  • Linux (Redhat7)
  • Deployed docker (cAdvisor, Prometheus, Grafana)
  • cAdvisor collect the metrics > Pass to Prometheus > Display with Grafana
  • Apache reverse proxy is in the environment (Therefore no direct connection with specific ports)

Issue:

  • cAdvisor does not show all container's uptime
  • Grafana does not show Prometheus and Grafana container's uptime
  • Only displays cAdvisor container's uptime

What I have (Issue): enter image description here enter image description here

What I want to have: enter image description here

Setting in Prometheus: enter image description here enter image description here

Command to run cAdvisor:

sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/dev/disk/:/dev/disk:ro --publish=8080:8080 --name=cadvisor --detach=true --privileged=true --volume=/cgroup:/cgroup:ro --network=docker8443 --ip=172.28.5.201 google/cadvisor:latest

Questions:

  • Is something missing in Prometheus target to show all container's uptime?
  • Is something missing in cAdvisor setup?
  • Is query for the graph incorrect? (I have tried following, but it shows "No data point") enter image description here
2
linuxdockergrafanaprometheuscadvisor

2 Answers

2
votes

I think the issue is about cadvisor. You can try check the cadvisor metrics endpoint: http://172.28.5.201:8080/metrics to make sure the cadvisor return the metrics samples:

example:

# HELP container_start_time_seconds Start time of the container since unix epoch in seconds.
# TYPE container_start_time_seconds gauge
container_start_time_seconds{id="/",image="",name=""} 1.525939343e+09
container_start_time_seconds{id="/docker",image="",name=""} 1.526006565e+09
container_start_time_seconds{id="/docker/d4b87911bd0842ee1d6969e6a05aa3d36a48a801184faf14e1b23169e056da92",image="busybox",name="trusting_bassi"}
0
votes

On top of Yunlong's answer: add --volume=/sys:/sys:ro You will also need to run following under shell:

$ mount -o remount,rw '/sys/fs/cgroup'

$ ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu

Otherwise your cAdvisor crashes immediately after container is up.