Kubernetes heapster excessive metric reporting on GKE

Question

I run a small 3-node kubernetes cluster on Google Container Engine (GKE) with metrics logging to Google Cloud Monitoring (GCM). Recently I noticed that metrics stopped being reported to GCM. The heapster log showed that requests were rejected due to GCM quota being exceeded.

Checking the API manager on Google Developers Console, I saw usage in excess of the 50000 req/day quota.

Now, each time I start the heapster pod, it hammers the GCM sink with 2-5 req/s in average.

I tried increasing heapster sink_frequency parameter, but that only results in more request, less frequently, which adds up to the same amount over time.

The master and nodes are all on latest Kubernetes version 1.1.3.

Anyone seen this on GKE? What can I do to restore sane levels of requests to GCL? At the moment I only get a few hours worth of metrics per day before the quota is exhausted.

Answer 1

The answer depends a bit on what your setup is, but it sounds like you're trying to set up Heapster on your own, with the Google Cloud Monitoring ("gcm") sink enabled. This setup has definite issues with using up quota, and isn't recommended for GKE users. You could reduce the number of requests a bit by increasing the stats_resolution option so that there will be less data to push to GCM, but you'd be best off just switching to GKE's built-in GCM integration.

You can enable the built-in integration by removing your custom-configured Heapster pod/replication controller from the cluster, then ensuring that the cloud monitoring is enabled for your cluster by running gcloud alpha container clusters update CLUSTERNAME --monitoring-service=monitoring.googleapis.com. You should then be able to find your metrics in the UI (after a one-time delay of up to an hour) as described in this recent blog post on GKE and GCM.