Kubernetes prometheus metrics for running pods and nodes?

Question

I've set up prometheus to monitor kubernetes metrics by following the prometheus documentation.

A lot of useful metrics now show up in prometheus.

However, I can't see any metrics referencing the status of my pods or nodes.

Ideally - I'd like to be able to graph the pod status (Running, Pending, CrashLoopBackOff, Error) and nodes (NodeReady, Ready).

Is this metric anywhere? If not, can I add it somewhere? And how?

Answer 1

The regular kubernetes setup does not expose these metrics - further discussion here.

However, another service can be used to collect these cluster level metrics: https://github.com/kubernetes/kube-state-metrics.

This currently provides node_status_ready and pod_container_restarts which sound like what I want.

Answer 2

I don't think such metrics exist.

You have to modify the source code to add them. Take a look at this file on how to register a metric: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/metrics.go, and take a look at this line on how to record a metric: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pleg/generic.go#L180

Answer 3

I've found that I can monitor these metrics using heapster & snap, which is a plausible workaround for my case. Let me know if that's something you're also using and I'll give you the proper metrics to get this data.