Kubernetes Deployment Rolling Updates

1
votes

I have an application that I deploy on Kubernetes.

This application has 4 replicas and I'm doing a rolling update on each deployment.

This application has a graceful shutdown which can take tens of minutes (it has to wait for running tasks to finish).

My problem is that during updates, I have over-capacity since all the older version pods are stuck at "Terminating" status while all the new pods are created.

During the updates, I end up running with 8 containers and it is something I'm trying to avoid.

I tried to set maxSurge to 0, but this setting doesn't take into consideration the "Terminating" pods, so the load on my servers during the deployment is too high.

The behaviour I'm trying to get is that new pods will only get created after the old version pods finished successfully, so at all times I'm not exceeding the number of replicas I set.

I wonder if there is a way to achieve such behaviour.

2
kuberneteskubernetes-deployment
With maxSurge you are saying to kubernetes how many pod you allow to live about the desired count. Have you tried setting maxUnavailable to 1 ? It's an amount of pods that can be unavailable during the update process. - acid_fuji
That won't cut it since each deploy will take a few hours since it will replace the pods one by one. - Shachar Ashkenazi

2 Answers

1
votes

What I ended up doing is creating a StatefulSet with podManagementPolicy: Parallel and updateStrategy to OnDelete.

I also set terminationGracePeriodSeconds to the maximum time it takes for a pod to terminate.

As a part of my deployment process, I apply the new StatefulSet with the new image and then delete all the running pods.

This way all the pods are entering Terminating state and whenever a pod finished its task and terminated a new pod with the new image will replace it.

This way I'm able to keep a static number of replicas during the whole deployment process.

0
votes

Let me suggest the following strategy:

  1. Deployments implement the concept of ready pods to aide rolling updates. Readiness probes allow the deployment to gradually update pods while giving you the control to determine when the rolling update can proceed.

  2. A Ready pod is one that is considered successfully updated by the Deployment and will no longer count towards the surge count for deployment. A pod will be considered ready if its readiness probe is successful and spec.minReadySeconds have passed since the pod was created. The default for these options will result in a pod that is ready as soon as its containers start.

So, what you can do, is implement (if you haven't done so yet) a readiness probe for your pods in addition to setting the spec.minReadySeconds to a value that will make sense (worst case) to the time that it takes for your pods to terminate.

This will ensure rolling out will happen gradually and in coordination to your requirements.

In addition to that, don't forget to configure a deadline for the rollout. By default, after the rollout can’t make any progress in 10 minutes, it’s considered as failed. The time after which the Deployment is considered failed is configurable through the progressDeadlineSeconds property in the Deployment spec.