1
votes

My GKE clusters are not monitored by Stackdriver, despite having Cloud Monitoring enabled. In particular https://app.google.stackdriver.com/gke simply reports "You do not have any resources of this type being monitored by Stackdriver".

  • Clusters running Kubernetes 1.2.4, created in mid-May 2016.
  • Some had Cloud Monitoring enabled on creation, some had it enabled via gcloud container clusters update serve --monitoring-service=monitoring.googleapis.com.
  • All clusters are running the heapster pod with 2/2 ready, status of Running, no restarts e.g. as heapster-v1.0.2-594732231-76sdj.
  • All clusters are running the heapster service with a cluster IP, no external IP and port 80/TCP.
  • Logs of heapster pod, heapster container look like:

I0609 09:15:05.000189 1 manager.go:79] Scraping metrics start: 2016-06-09 09:14:00 +0000 UTC, end: 2016-06-09 09:15:00 +0000 UTC

I0609 09:15:05.023729 1 manager.go:152] ScrapeMetrics: time: 23.435264ms size: 25

  • Logs of heapster pod, heapster-nanny container look like:

I0609 09:18:03.968201 1 nanny_lib.go:90] The number of nodes is 1

I0609 09:18:03.971459 1 nanny_lib.go:98] The container resources are &{map[memory:{213909504.000000000 BinarySI} cpu:{0.100000000 DecimalSI}] map[cpu:{0.100000000 DecimalSI} memory:{213909504.000000000 BinarySI}]}

I0609 09:18:03.971522 1 nanny_lib.go:102] The expected resources are &{map[cpu:{0.100000000 DecimalSI} memory:{218103808.000000000 BinarySI}] map[cpu:{0.100000000 DecimalSI} memory:{218103808.000000000 BinarySI}]}

I0609 09:18:03.971634 1 nanny_lib.go:106] Resources are within the expected limits.

1

1 Answers

2
votes

it turns out that Stackdriver had a problem of not properly displaying new-ish clusters in its UI. Metrics were being properly collected and were available via their API the entire time. The problem should now be resolved, but the team is very sorry for any trouble this caused and will work to catch problems like this more quickly.