Kubernetes pod recognizes if he will be shut down

Question

I have a simple docker container, that get jobs from a queue. A scaling script handles the amount of pods that run right now by sending an api request to the replication controller.

The problem is that I don't want to shut down a pod when he is doing a job right now. If I reduce the amount of pods kubernetes will delete the last created pod.

So I want to know if it's possible to tell the last pod, that he will be shut down in some minutes so he should not get a new job? One solution would be that the scaling script is is calling the last pod manually. But is it possible for every pod to listen to for the shut down event for his own instance?

Answer 1

You can specify a grace period pods are given to complete their work. Pods are sent a TERM signal when they should start to shutdown. Your pods could listen for that, complete their current job, and not take another but exit instead. The grace period defaults to 30 seconds, but you could set that to be long enough for the typical job to complete via the terminationGracePeriodSeconds field in the pod spec.

See https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods for details