Connection Interrupted during capistrano migration

Question

I just ran into an issue while running a production deploy using capistrano. We had just finished a large refactor that contained a lot of database migrations.

During the deploy, the worst thing possible happened and my ssh connection dropped, while cap was running through the migrations.

I think there's an issue with sshing through our load balancer but thats beside the point.

I managed to get the migrations to fully run by running screen on the server, migrating here, and deploying afterward.

The app is now up and seems to be working fine, but I was just wondering does anyone know how capistrano handles migrations if the connection is interrupted?

Can I be sure the the migration executing when the connection dropped was completed?

What are the chances of half performed or migrations performed twice?

I would assume that cap wraps each migration in a db transaction that would rollback if an error occurred, would this be the case?

Answer 1

Loosing the SSH just means you lost access to the instance, but the instance and command were already (and still) running.

Capistrano will only be executing rake db:migrate under the hood, which basically means you can rely on the migrations being fully run to completion or an error found. Even if the process is halted, since after each migration the schema is changed the migration won't be run again. And if it is halted, then the schema doesn't change, prompting the migration to be run again when you (or capistrano) runs rake db:migrate another time.

If you want to always be sure no problems arise (or can be effectively dealt with) write migrations that are perfectly reversible - this ensures that on a migration error rake can rollback previously run migrations (running on the same sequence that invoked the error). Usually most migrations will be reversible by default with no extra code, but if you have more complex behaviour they might need a particular reverse method to be written (say you're changing columns and casting values into something else). The only thing that by default won't be reversible is table/columns drops, unless you add the type to the remove statement, since it's optional on a drop. If column type is present then rails will know how to revert the drop as well (without needing a specific reverse method).