unicorn using old release executable (rbenv + runit)

Question

I am running into an issue that should be easy to fix but I can't seem to find the magic.

Unicorn is using runit and attempting to point to the current releases Gemfile. At the end of each deploy, unicorn is sent a USR2 signal and the old master is successfully replaced using the "zero downtime deploy" method. This was all fine for a while, until I hit the point where old releases were being cleaned out. It was then I realized runit was pointing to the release directory that was used when the unicorn master was first created.

I have tried adding this block to unicorn.rb:

before_exec do |server|
    ENV['BUNDLE_GEMFILE'] = "/home/deploy/app/current/Gemfile"
end

I have also tried wrapping bundler with this script:

#!/bin/bash

source /etc/profile.d/rbenv.sh

bundle $@

I have even tried setting the BUNDLE_GEMFILE env variable manually, but none of these seem to work.

When I take a look at the unicorn stdout after deploying I see this:

executing ["/home/deploy/app/releases/1279f2e2d88c90ba4e02eaba611a4ee6de6fee77/vendor/bundle/ruby/2.0.0/bin/unicorn", "-E", "staging", "-c", "/home/deploy/app/shared/config/unicorn.rb", "-D", {12=>#<Kgio::UNIXServer:fd 12>}] (in /home/deploy/app/releases/d6a582935d84cc0bec2e760c14d804a7c5e2146c)

As you can see, the command is being run in the new release directory (d6a582935d84cc0bec2e760c14d804a7c5e2146c), but the unicron executable is pointing to an old release (1279f2e2d88c90ba4e02eaba611a4ee6de6fee77).

If I manually stop/start unicorn, then the correct release is used, but that would mean not having "zero downtime deploys".

I'm not sure what the issue is, but I figure it is something to do with rbenv paths. Anybody have suggestions for how to get unicorn pointing to the correct (current) release?

I am using the application_ruby LWRP with Chef to deploy if anyone is familiar. It basically just emulates Capistrano.

Relevant configs:

unicorn.rb:

current_path = '/home/deploy/app/current'

working_directory '/home/deploy/app/current'

worker_processes 8

listen "/home/deploy/app/shared/sockets/cms.sock"

timeout 60

pid '/home/deploy/app/shared/pids/unicorn.pid'

stderr_path '/home/deploy/app/shared/log/unicorn/unicorn.stderr.log'
stdout_path '/home/deploy/app/shared/log/unicorn/unicorn.stderr.log'

preload_app true

# Enable Copy on Write Garbage Collector - http://www.rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end

  old_pid = "/home/deploy/app/shared/pids/unicorn.pid.oldbin"

  if File.exists?(old_pid) && server.pid != old_pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :TERM : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
  # sleep 0 for small app, sleep 2 for big (300mb+)
  sleep 1
end

before_exec do |server|
  ENV['BUNDLE_GEMFILE'] = "#{current_path}/Gemfile"
end

after_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end
end

runit unicorn script:

#!/bin/bash

if [ -d "/home/deploy/app/current" ] ; then

    function is_unicorn_alive {
        set +e
        if [ -n $1 ] && kill -0 $1 >/dev/null 2>&1; then
            echo "yes"
        fi
        set -e
     }

    echo "Service PID: $$"

    CUR_PID_FILE="/home/deploy/app/shared/pids/unicorn.pid"
    OLD_PID_FILE=$CUR_PID_FILE.oldbin

    if [ -e $OLD_PID_FILE ]; then
        OLD_PID=$(cat $OLD_PID_FILE)
        echo "Waiting for existing master ($OLD_PID) to exit"
        while [ -n "$(is_unicorn_alive $OLD_PID)" ]; do
            /bin/echo -n '.'
            sleep 2
        done
    fi

    if [ -e $CUR_PID_FILE ]; then
        CUR_PID=$(cat $CUR_PID_FILE)
        if [ -n "$(is_unicorn_alive $CUR_PID)" ]; then
            echo "Unicorn master already running. PID: $CUR_PID"
            RUNNING=true
        fi
    fi

    if [ ! $RUNNING ]; then
        echo "Starting unicorn"
        cd /home/deploy/app/current
        chown deploy:deploy /home/deploy/app/shared/pids/

        chpst -u deploy \
        /opt/rbenv/shims/bundle exec \
        unicorn \
        -E staging -c /home/deploy/app/shared/config/unicorn.rb -D

        sleep 3
        CUR_PID=$(cat $CUR_PID_FILE)
    fi

    function restart {
        echo "Initialize new master with USR2"
        kill -USR2 $CUR_PID
        # Make runit restart to pick up new unicorn pid
        sleep 2
        echo "Restarting service to capture new pid"
        exit
    }

    function graceful_shutdown {
        echo "Initializing graceful shutdown"
        kill -QUIT $CUR_PID
    }

    function unicorn_interrupted {
        echo "Unicorn process interrupted. Possibly a runit thing?"
    }

    trap restart HUP QUIT USR2 INT
    trap graceful_shutdown TERM KILL
    trap unicorn_interrupted ALRM

    echo "Waiting for current master to die. PID: ($CUR_PID)"
    while [ -n "$(is_unicorn_alive $CUR_PID)" ]; do
        /bin/echo -n '.'
        sleep 2
    done
else
    sleep 1
fi

shlant- shlant- · Accepted Answer · 2014-09-15T19:54:37

Found it!

Added this to unicorn.rb:

Unicorn::HttpServer::START_CTX[0] = "#{current_path}/bin/unicorn"

unicorn using old release executable (rbenv + runit)

1 Answers