Fetching Stackdriver Monitoring TimeSeries data for a pod running on a k8s cluster on GKE using the REST API

Question

My objective is to fetch the time series of a metric for a pod running on a kubernetes cluster on GKE using the Stackdriver TimeSeries REST API.

I have ensured that Stackdriver monitoring and logging are enabled on the kubernetes cluster.

Currently, I am able to fetch the time series of all the resources available in a cluster using the following filter:

metric.type="container.googleapis.com/container/cpu/usage_time" AND resource.labels.cluster_name="<MY_CLUSTER_NAME>"

In order to fetch the time series of a given pod id, I am using the following filter:

metric.type="container.googleapis.com/container/cpu/usage_time" AND resource.labels.cluster_name="<MY_CLUSTER_NAME>" AND resource.labels.pod_id="<POD_ID>"

This filter returns an HTTP 200 OK with an empty response body. I have found the pod ID from the metadata.uid field received in the response of the following kubectl command:

kubectl get deploy -n default <SERVICE_NAME> -o yaml

However, when I use the Pod ID of a background container spawned by GKE/Stackdriver, I do get the time series values.

Since I am able to see Stackdriver metrics of my pod on the GKE UI, I believe I should also get the metric values using the REST API.

My doubts/questions are:

1. Am I fetching the Pod ID of my pod correctly using kubectl?
2. Could there be some issue with my cluster setup/service deployment due to which I'm unable to fetch the metrics?
3. Is there some other way in which I can get the time series of my pod using the REST APIs?

Answer 1

1. I wouldn't rely on kubectl get deploy for pod ids. I would get them with something like kubectl -n default get pods | grep <prefix-for-your-pod> | awk '{print $1}'

2. I don't think so, but the best way to find out is opening a support ticket with GCP if you have any doubts.

3. Not that I'm aware of, Stackdriver is the monitoring solution in GCP. Again, you can check with GCP support. There are other tools that you can use to get metrics from Kubernetes like Prometheus. There are multiple guides on the web on how to set it up with Grafana on k8s. This is one for example.

Hope it helps!

Answer 2

Am I fetching the Pod ID of my pod correctly using kubectl?

You could use JSONpath as output with kubectl, in this case iterating over the Pods and fetching the metadata.name and metadata.uid fields:

kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\n"}{end}'

which will output something like this:

nginx-65899c769f-2j775  d4fr5t6-bc2f-11e8-81e8-42010a84011f
nginx2-77b5c9d48c-7qlps 4f5gh6r-bc37-11e8-81e8-42010a84011f

Could there be some issue with my cluster setup/service deployment due to which I'm unable to fetch the metrics?

As @Rico mentioned in his answer, contacting the GCP support could be a way forward if you don't get further with the troubleshooting, see below.

Is there some other way in which I can get the time series of my pod using the REST APIs?

You could use the APIs Explorer or the Metrics Explorer from within the Stackdriver portal. There's some good troubleshooting tips here with a link to the APIs Explorer. In the Stackdriver Metrics Explorer it's fairly easy to reassemble the filter you've used using dropdown lists to choose e.g. a particular pod_id.

Taken from the Troubleshooting the Monitoring guide (linked above) regarding an empty HTTP 200 response on filtered queries:

If your API call returns status code 200 and an empty response, there are several possibilities:

• If your call uses a filter, then the filter might not have matched anything. The filter match is case-sensitive. To resolve filter problems, start by specifying only one filter component, such as metric.type, and see if you get results. Add the other filter components one-by-one.
• If you are working with a custom metric, you might not have specified the project where your custom metric is defined.*

I found this link when reading through the documentation of the Monitoring API. That link will get you to the APIs Explorer with some pre-filled fields, change these accordingly and add your own filter.

I have not tested more using the REST API at the moment but hopefully this could get you forward.