Kubernetes Pod Termination doesn't happen immediately, have to wait until grace period expires

Question

I have a helm chart that has one deployment/pod and one service. I set the deployment terminationGracePeriodSeconds to 300s. I didn't have any pod lifecycle hook, so if I terminate the pod, the pod should terminate immediately. However, now the pod will determine until my grace period ends!

Below is the deployment template for my pod:

$ kubectl get pod hpa-poc---jcc-7dbbd66d86-xtfc5 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  creationTimestamp: "2021-02-01T18:12:34Z"
  generateName: hpa-poc-jcc-7dbbd66d86-
  labels:
    app.kubernetes.io/instance: hpa-poc
    app.kubernetes.io/name: -
    pod-template-hash: 7dbbd66d86
  name: hpa-poc-jcc-7dbbd66d86-xtfc5
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: hpa-poc-jcc-7dbbd66d86
    uid: 66db29d8-9e2d-4097-94fc-b0b827466e10
  resourceVersion: "127938945"
  selfLink: /api/v1/namespaces/default/pods/hpa-poc-jcc-7dbbd66d86-xtfc5
  uid: 82ed4134-95de-4093-843b-438e94e408dd
spec:
  containers:
  - env:
    - name: _CONFIG_LINK
      value: xxx
    - name: _USERNAME
      valueFrom:
        secretKeyRef:
          key: username
          name: hpa-jcc-poc
    - name: _PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: hpa-jcc-poc
    image: xxx
    imagePullPolicy: IfNotPresent
    name: -
    resources:
      limits:
        cpu: "2"
        memory: 8Gi
      requests:
        cpu: 500m
        memory: 2Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-hzmwh
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: xxx
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-hzmwh
    secret:
      defaultMode: 420
      secretName: default-token-hzmwh
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-02-01T18:12:34Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-02-01T18:12:36Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-02-01T18:12:36Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-02-01T18:12:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://c4c969ec149f43ff4494339930c8f0640d897b461060dd810c63a5d1f17fdc47
    image: xxx
    imageID: xxx
    lastState: {}
    name: -
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2021-02-01T18:12:35Z"
  hostIP: 10.0.35.137
  phase: Running
  podIP: 10.0.21.35
  qosClass: Burstable
  startTime: "2021-02-01T18:12:34Z"

When I tried to terminate the pod (I used helm delete command), you can see in the time, it terminated after 5 min which is the grace period time.

$ helm delete hpa-poc
release "hpa-poc" uninstalled
$ kubectl get pod -w | grep hpa
hpa-poc-jcc-7dbbd66d86-xtfc5         1/1     Terminating   0          3h10m
hpa-poc-jcc-7dbbd66d86-xtfc5         0/1     Terminating   0          3h15m
hpa-poc-jcc-7dbbd66d86-xtfc5         0/1     Terminating   0          3h15m

So I suspect it's something for my pod/container configuration issue. Because I have tried with the other simple Java App deployment and it can terminate immediately once I terminate the pod.

BTW, I am using AWS EKS Cluster. Not sure its AWS specific as well.

So any suggestions?

If you are pushing container/application/pod logs to any logging system, then what do you see in logs of such delayed terminated container/application/pod? — amitd
I'm more inclined to the idea that your Java application is unable to catch the SIGTERM sent by kubelet rather than there is an issue with Pod or Container. That's why it's killed after 300 seconds (when SIGKILL is sent). You can read more about it for example here: cloud.google.com/blog/products/gcp/… — Dawid Kruk

Tianbing Leng Tianbing Leng · Accepted Answer · 2021-02-03T01:12:07

I find the issue. When I exec into the container, I noticed there is one process running, which is the tailing log process.

So, I need to kill the process and add that into the prestop hook. After that, my container can shut down immediately.

Kubernetes Pod Termination doesn't happen immediately, have to wait until grace period expires

1 Answers