How can I track the time pods are in Terminating status?

Question

I am developing a shell script which will forcefully delete pod stuck in Terminating status(due to any reason).

kubectl get pods --all-namespaces | grep Terminating | while read line; do 
pod_name=$(echo $line | awk '{print $2}' ) name_space=$(echo $line | awk 
'{print $1}' ); kubectl delete pods $pod_name -n $name_space --grace-period=0 --force; 
done

But to be on safe side I want to delete pod only if its in Terminating status since last 10 minute.

How can I get this time as kubectl get pods --all-namespaces | grep Terminating give AGE of pod not time of current status.

As per documentation pod-lifecycle status.phase can have only 5 different values and none of them is Terminating. This means that termination state in not reflected in this field and therefore filtering by the phase field will not help.

Update1: After suggestions from @Vitalii following is output, These pods (statefulset) are in terminating status due to node shutdown

kubectl -n my_ns get pod mypod-0  -o json| jq '.status.containerStatuses[]'
{
  "containerID": "docker://bc4fdf965b19dffd5cb54dddb1931d9f522aad885d1576d03a41cd45d6d60c0d",
  "image": "mypod:1.11.0",
  "imageID": "docker://sha256:128b5cb1a95cfa001620d9e54b8cf9d0ef96b2a261cab4850accab4cfa0fe6cb",
  "lastState": {},
  "name": "mypod",
  "ready": true,
  "restartCount": 0,
  "started": true,
  "state": {
    "running": {
      "startedAt": "2021-04-21T16:42:23Z"
    }
  }
}

VASャ VASャ · Accepted Answer · 2021-05-04T09:11:06

Termination date-time is saved in Pod's property .metadata.deletionTimestamp:

kubectl get pods -o jsonpath="{.items[*].metadata.deletionTimestamp}"

or for a single pod:

kubectl get pod pod_name -o jsonpath="{.metadata.deletionTimestamp}"

Note: properties .metadata.deletionTimestamp and .metadata.deletionGracePeriodSeconds don't exist for pods in Ready state.

Example:

k8s-m1:~$ kubectl get nodes
NAME     STATUS     ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
k8s-m1   Ready      master   256d   v1.18.8   10.156.0.49   <none>        Ubuntu 18.04.5 LTS   5.4.0-1040-gcp   docker://19.3.12
k8s-w1   NotReady   <none>   256d   v1.18.8   10.156.0.50   <none>        Ubuntu 18.04.5 LTS   5.4.0-1040-gcp   docker://19.3.12

$ kubectl get pods
NAME                                          READY   STATUS        RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
aloha-pod                                     1/1     Running       0          11d     192.168.42.159   k8s-m1   <none>           <none>
hello-pod                                     1/1     Terminating   0          11d     192.168.228.80   k8s-w1   <none>           <none>


k8s-m1:~$ kubectl get pod aloha-pod -o jsonpath="deletionTimestamp={.metadata.deletionTimestamp},  deletionGracePeriodSeconds={.metadata.deletionGracePeriodSeconds}";echo
deletionTimestamp=,  deletionGracePeriodSeconds=


k8s-m1:~$ kubectl get pod hello-pod -o jsonpath="deletionTimestamp={.metadata.deletionTimestamp},  deletionGracePeriodSeconds={.metadata.deletionGracePeriodSeconds}";echo
deletionTimestamp=2021-04-29T09:39:03Z,  deletionGracePeriodSeconds=30

How can I track the time pods are in Terminating status?

2 Answers