Monitoring pod termination time with prometheus

Question

I'm looking for a Prometheus metric that would allow me to monitor the time pods spend in the terminating state before vanishing into the void.

I've tried playing around with kube_pod_container_status_terminated but it only seems to register pods once they finish the termination process, but don't help me understand how long does it take to terminate a pod.
I've also looked at kube_pod_status_phase which I found out about in this channel a while ago but it also seems to lack this insight.

I'm currently collecting metrics on my k8s workload using cAdvisor, kube-state-metrics and the prometheus node-exporter, but would happily considering additional collectors if they contain the desired data.
A non-prometheus solution would also be great.
Any ideas? Thanks!

What version of Kubernetes do you use? You can collect this timestamps from API server. but solution different for versions. — Nick Rak
1.8.8, I'm aware of the option of an API request which can be parsed and then written as a metric but I was really hoping I can just get this info our of the myriad of metrics exported to prometheus. Thanks though. — Yaron Idan

VASャ VASャ · Accepted Answer · 2018-12-17T15:03:06

Kubernetes itself, Heapster and metrics-server don't provide such metrics, but you can get metrics close to what you've mentioned by installing kube-state-metrics. It has several pod metrics that reflect pods state:

kube_pod_status_phase
kube_pod_container_status_terminated
kube_pod_container_status_terminated_reason
kube_pod_container_status_last_terminated_reason

You can find the full list of pods metrics, provided by kube-state-metrics in Documentation.

There is also Bitnami Helm chart that could simplify the installation of kube-state-metrics.

Monitoring pod termination time with prometheus

1 Answers