Kubernetes rolling deployments and database migrations

Question

When processing a rolling update with database migrations, how does kubernetes handle this?

For an instance - I have an app that gets updated from app-v1 to app-v2, which includes a migration step to alter an existing table. So this would mean it requires me to run something like db:migrate for a rails app once deployed.

When a rolling deployment takes place on 3 replica set. It will deploy from one pod to another. Potentially allowing PODs that don't have the new version of the app to break.

Although this scenario is not something that happens very often. It's quite possible that it would. I would like to learn about the best/recommended approaches for this scenario.

Answer 1

One way to prevent an old version from breaking is to split a migration into multiple steps.

E.g. you want to rename a column in the database. Renaming the column directly would break old versions of the app. This can be split into multiple steps:

• Add a db migration that inserts the new column
• Change the app so that all writes go to the old and new column
• Run a task that copies all values from the old to the new column
• Change the app that it reads from the new column
• Add a migration that remove the old column

This is unfortunately quite a hassle, but prevents having a downtime with a maintenance page up.

Answer 2

Kubernetes does NOT natively handle rolling updates with db migrations. This is application specific, so the application developer will have to handle it. You may have to do the same things as you would do in a non-k8s setting.

Anyway, the way I would do it is:

• Scale your replicas to 1.
• Update the image.
• Scale your replicas to 3.

This is not fool-proof but doesn't involve code change. There is a small window between the db:migrate step and the actual server listening where a request will go to the older replica (which will terminate as soon as the new replica is ready). That request may or may not fail depending on whether the code block is directly related to the schema change.

If I didn't care much about downtime, then I would just use the recreate strategy.

Answer 3

I was solving this issue recently, and here is my way:

• Use deployment annotations to store the commands you normally need to run before or after the deployment.
• Create a script that will be able to read your deployment by name and then create a job to run the commands specified in the deployment annotations.
• When pushing an image to a docker registry add a webhook that will call your script specified in the previous point.
• To avoid problems with an incompatible DB structure:
  • Do not modify the db columns in a backwards incompatible way.
  • Do not remove the unused columns immediately in your migrations. You can do it in the next release. This way you will have only one migration script that you will run before the deployment.

P.S. To be able to work with Kubernetes in your script, you might want to get acquainted with these links: Kubernetes API, Kubernetes API Overview, Access Clusters Using the Kubernetes API, Client Libraries.