Does a Kubernetes rolling-update gracefully remove pods from a service load balancer

Question

Standard practice for a rolling update of hosts behind load balancer is to gracefully take the hosts out of rotation. This can be done by marking the host "un-healthy" and ensuring the host is no longer receiving requests from the load balancer.

Does Kubernetes do something similar for pods managed by a ReplicationController and servicing a LoadBalancer Service?

I.e., does Kubernetes take a pod out of the LoadBalancer rotation, ensure incoming traffic has died-down, and only then issue pod shutdown?

Answer 1

Actually, once you delete the pod, it will be in "terminating" state until it is destroyed (after terminationGracePeriodSeconds) which means it is removed from the service load balancer, but still capable of serving existing requests.

We also use "readiness" health checks, and preStop is synchronous, so you could make your preStop hook mark the readiness of the pod to be false, and then wait for it to be removed from the load balancer, before having the preStop hook exit.

Answer 2

Not quite. Kubernetes will send a stop command to the containers in the pod. If the application doesn't stop it will force kill the container (after terminationGracePeriodSeconds parameter).

There are a bunch of bugs opened to take care of this: https://github.com/kubernetes/kubernetes/issues/2789

I can't think of anything elegant that will do this.

There is a preStop parameter for pods that will execute a script before termination. You could modify from here the pod label and rename it to something else. This will fool the replication controller and it will see that it has now a lower number of replicas.

For the pods with this label you will have to make your own logic on stopping them when they have finished working.