Restart Unicorn with a USR2 - quitting old master

Question

So sending USR2 to Unicorn is awesome -- it starts up a new master with a new copy of your code and automatically picks up any changes. Sweet. My question is: how do I stop the old master? The apparently accepted way is in a before_fork:

before_fork do |server,worker|
  old_pid = '/var/www/current/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

The problem with this is that as soon as the new master (and new workers) are spawned, they kill the old master. So any requests to the site just sit there waiting for the new worker to start, usually for several seconds while the entire Rails stack loads.

If I remove my before_fork everything works as I would hope (from the client's point of view): I can reload my browser all day and every request is filled quickly, there's no indication of when the new master takes over (other than seeing my code changes appear now). But, the old master now hangs around until I manually send it a QUIT.

As far as I know there is no callback once a worker is done loading and ready to serve clients. That's really the callback I'm looking for. I could always create an initializer in Rails itself that looks for an old master and kills it, but that makes my heart hurt just thinking about it.

There must be a way!

Answer 1

I partially solved this: the behavior I'm seeing is caused by not using preload_app true. If you have this set then the entire app is loaded by the master and workers are very fast to spawn. So if the first worker kills the old master at this point it's okay because said worker can start serving requests immediately!

If you can't use preload_app true then your best bet is probably to move that old-pid-quit behavior into a Rails initializer so that the first worker that brings up your app can kill the old master once Rails has started up and is ready to serve requests.

Answer 2

It appears that sending a HUP signal to the Unicorn master is a better alternative if preload_app is false.

From http://unicorn.bogomips.org/SIGNALS.html:

HUP - reloads config file and gracefully restart all workers. If the “preload_app” directive is false (the default), then workers will also pick up any application code changes when restarted.

Answer 3

Here is what I have in my before_fork block:

  old_pid = "#{server.config[:pid]}.oldbin"
  if old_pid != server.pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end

To restart my unicorn I have a bash script that has a method like this:

if sig USR2 && sig 0 && oldsig QUIT
then
    n=$TIMEOUT
    while test -s $old_pid && test $n -ge 0
    do
        printf '.' && sleep 1 && n=$(( $n - 1 ))
    done
    echo

    if test $n -lt 0 && test -s $old_pid
    then
        echo >&2 "$old_pid still exists after $TIMEOUT seconds"
        exit 1
    fi
    exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
$CMD
;;

The bash script sends a USR2 signal, which forks the new unicorn and creates the old pid. Then it sends the old unicorn the QUIT signal using the old pid.

This process works really well and was taken from Where Unicorns go to die: Watching unicorn workers with monit, which is an excellent resource.