Unicorn with Capistrano restart doesn't work

Question

I have two Rails4-Apps set up on my server with Capistrano 2.15, Unicorn and Nginx.

When I do cap:deploy, I get this error:

executing "/etc/init.d/unicorn_myapp1 restart"
servers: ["123.45.678.910"]
[123.45.678.910] executing command
** [out :: 123.45.678.910] Couldn't reload, starting 'cd/home/deployer/apps/myapp1/current; bundle exec unicorn -D -c /home/deployer/apps/myapp1/current/config/unicorn.rb -E production' instead
** [out :: 123.45.678.910] 
** [out :: 123.45.678.910] master failed to start, check stderr log for details
command finished in 1845ms
failed: "rvm_path=$HOME/.rvm $HOME/.rvm/bin/rvm-shell 'ruby-2.0.0-p353' -c '/etc/init.d/unicorn_myapp1 restart'" on 123.45.678.910

Further Information:

If I kill the workers manually on the server, the deployment process works (but the restart error still shows up).
I suspect, the error has something to do with the fact, that I'm running 2 Apps simultaneously. If I'm running only one app it works.

Output for finalize_update step:

* 2014-01-13 13:31:11 executing `deploy:finalize_update'
    triggering before callbacks for `deploy:finalize_update'
  * 2014-01-13 13:31:11 executing `deploy:assets:symlink'
  * executing "rm -rf /home/deployer/apps/myapp1/releases/20140113123110/public/assets && mkdir -p /home/deployer/apps/myapp1/releases/20140113123110/public && mkdir -p /home/deployer/apps/myapp1/shared/assets && ln -s /home/deployer/apps/myapp1/shared/assets /home/deployer/apps/myapp1/releases/20140113123110/public/assets"
    servers: ["xxx.xxx.xxx.xxx"]
    [xxx.xxx.xxx.xxx] executing command
    command finished in 816ms
  * 2014-01-13 13:31:12 executing `bundle:install'
  * executing "cd /home/deployer/apps/myapp1/releases/20140113123110 && bundle install --gemfile /home/deployer/apps/myapp1/releases/20140113123110/Gemfile --path /home/deployer/apps/myapp1/shared/bundle --deployment --quiet --without development test"
    servers: ["xxx.xxx.xxx.xxx"]
    [xxx.xxx.xxx.xxx] executing command
    command finished in 2048ms
  * executing "chmod -R -- g+w /home/deployer/apps/myapp1/releases/20140113123110 && rm -rf -- /home/deployer/apps/myapp1/releases/20140113123110/public/system && mkdir -p -- /home/deployer/apps/myapp1/releases/20140113123110/public/ && ln -s -- /home/deployer/apps/myapp1/shared/system /home/deployer/apps/myapp1/releases/20140113123110/public/system && rm -rf -- /home/deployer/apps/myapp1/releases/20140113123110/log && ln -s -- /home/deployer/apps/myapp1/shared/log /home/deployer/apps/myapp1/releases/20140113123110/log && rm -rf -- /home/deployer/apps/myapp1/releases/20140113123110/tmp/pids && mkdir -p -- /home/deployer/apps/myapp1/releases/20140113123110/tmp/ && ln -s -- /home/deployer/apps/myapp1/shared/pids /home/deployer/apps/myapp1/releases/20140113123110/tmp/pids"
    servers: ["xxx.xxx.xxx.xxx"]
    [xxx.xxx.xxx.xxx] executing command
    command finished in 512ms
    triggering after callbacks for `deploy:finalize_update'
  * 2014-01-13 13:31:15 executing `deploy:symlink_config'
  * executing "ln -nfs /home/deployer/apps/myapp1/shared/config/database.yml /home/deployer/apps/myapp1/releases/20140113123110/config/database.yml"
    servers: ["xxx.xxx.xxx.xxx"]
    [xxx.xxx.xxx.xxx] executing command
    command finished in 615ms
    triggering after callbacks for `deploy:update_code'
* 2014-01-13 13:31:15 executing `deploy:assets:precompile'

Nginx.conf:

upstream myapp.example.com {
  server unix:/tmp/unicorn.myapp.sock fail_timeout=0;
}

server {
  listen 80;
  server_name myapp.example.com;
  root /home/deployer/apps/myapp/current/public;

  location ~ ^/(assets)/  {
    root /home/deployer/apps/myapp/current/public;
    gzip_static on; # to serve pre-gzipped version
    expires max;
    add_header Cache-Control public;
  }

  try_files $uri/index.html $uri @myapp.example.com;
  location @myapp.example.com {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://myapp.example.com;
  }

  error_page 500 502 503 504 /500.html;
  client_max_body_size 4G;
  keepalive_timeout 10;
}

The Unicorn log shows this error:

/home/deployer/apps/myapp1/shared/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:219:in `pid=': Already running on PID:20755 (or $
        from /home/deployer/apps/myapp1/shared/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:142:in `start'
        from /home/deployer/apps/myapp1/shared/bundle/ruby/2.0.0/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>'
        from /home/deployer/apps/myapp1/shared/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
        from /home/deployer/apps/myapp1/shared/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

This is my unicorn.rb file:

root = "/home/deployer/apps/myapp1/current"
working_directory root
pid "#{root}/tmp/pids/unicorn-myapp1.pid"
stderr_path "#{root}/log/unicorn.log"
stdout_path "#{root}/log/unicorn.log"

listen "/tmp/unicorn.myapp1.sock"
worker_processes 2
timeout 30

# Force the bundler gemfile environment variable to
# reference the capistrano "current" symlink
before_exec do |_|
  ENV["BUNDLE_GEMFILE"] = File.join(root, 'Gemfile')
end

Capfile:

load 'deploy'
load 'deploy/assets'
load 'config/deploy'

This is my unicorn_init.sh:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          unicorn
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Manage unicorn server
# Description:       Start, stop, restart unicorn server for a specific application.
### END INIT INFO
set -e

# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/deployer/apps/myapp1/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="cd $APP_ROOT; bundle exec unicorn -D -c $APP_ROOT/config/unicorn.rb -E production"
AS_USER=deployer
set -u

OLD_PIN="$PID.oldbin"

sig () {
  test -s "$PID" && kill -$1 `cat $PID`
}

oldsig () {
  test -s $OLD_PIN && kill -$1 `cat $OLD_PIN`
}

run () {
  if [ "$(id -un)" = "$AS_USER" ]; then
    eval $1
  else
    su -c "$1" - $AS_USER
  fi
}

case "$1" in
start)
  sig 0 && echo >&2 "Already running" && exit 0
  run "$CMD"
  ;;
stop)
  sig QUIT && exit 0
  echo >&2 "Not running"
  ;;
force-stop)
  sig TERM && exit 0
  echo >&2 "Not running"
  ;;
restart|reload)
  sig HUP && echo reloaded OK && exit 0
  echo >&2 "Couldn't reload, starting '$CMD' instead"
  run "$CMD"
  ;;
upgrade)
  if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
  then
    n=$TIMEOUT
    while test -s $OLD_PIN && test $n -ge 0
    do
      printf '.' && sleep 1 && n=$(( $n - 1 ))
    done
    echo

    if test $n -lt 0 && test -s $OLD_PIN
    then
      echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
      exit 1
    fi
    exit 0
  fi
  echo >&2 "Couldn't upgrade, starting '$CMD' instead"
  run "$CMD"
  ;;
reopen-logs)
  sig USR1
  ;;
*)
  echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
  exit 1
  ;;
esac

And here is my deploy.rb:

require "bundler/capistrano"
require "rvm/capistrano"

set :rvm_ruby_string, 'ruby-2.0.0-p353'
set :rvm_type, :user

server "123.456.789.101", :web, :app, :db, primary: true

set :application, "myapp1"
set :user, "deployer"
set :deploy_to, "/home/#{user}/apps/#{application}"
set :deploy_via, :remote_cache
set :use_sudo, false

set :scm, "git"
set :repository, "[email protected]:myusername/#{application}.git"
set :branch, "master"


default_run_options[:pty] = true
ssh_options[:forward_agent] = true

before "deploy:restart", :symlink_directories
task :symlink_directories do
  run "ln -nfs #{shared_path}/uploads #{release_path}/public/uploads"
end

after "deploy", "deploy:cleanup" # keep only the last 5 releases

namespace :deploy do
  %w[start stop restart].each do |command|
    desc "#{command} unicorn server"
    task command, roles: :app, except: {no_release: true} do
      run "/etc/init.d/unicorn_#{application} #{command}"
    end
  end

  task :setup_config, roles: :app do
    sudo "ln -nfs #{current_path}/config/nginx.conf /etc/nginx/sites-enabled/#{application}"
    sudo "ln -nfs #{current_path}/config/unicorn_init.sh /etc/init.d/unicorn_#{application}"
    run "mkdir -p #{shared_path}/config"
    put File.read("config/database.example.yml"), "#{shared_path}/config/database.yml"
    puts "Now edit the config files in #{shared_path}."
  end
  after "deploy:setup", "deploy:setup_config"

  task :symlink_config, roles: :app do
    run "ln -nfs #{shared_path}/config/database.yml #{release_path}/config/database.yml"
  end

  after "deploy:finalize_update", "deploy:symlink_config"

  desc "Make sure local git is in sync with remote."
  task :check_revision, roles: :web do
    unless `git rev-parse HEAD` == `git rev-parse origin/master`
      puts "WARNING: HEAD is not the same as origin/master"
      puts "Run `git push` to sync changes."
      exit
    end
  end
  before "deploy", "deploy:check_revision"
end

master failed to start, check stderr log for details Did you check your log file for errors? — Erez Rabih
@ErezRabih it says, that PID is already running. I added the complete error to my question. — crispychicken

severin severin · Accepted Answer · 2014-01-08T12:00:23

You probably have a permission problem:

During deployment, capistrano tries to restart unicorn as using the init script. The init script tries to send the HUP signal to the master to signal it to restart its workers. If this fails, the script assumes that unicorn is not running and tries to start a new instance. When this happens, you get the error that you are experiencing (when the new instance tries to start, it notices the already running instance and terminates with the error you get).

So sending the HUP signal to the unicorn master seems to be failing in your case. I suspect this is due to a permission problem. Please check that unicorn is started by the same user that you are using during deploying (your config looks correct, using deployer everywhere, but maybe the unicorn process that is currently running was started by root or someone else?).

For troubleshooting, I suggest the following:

log into your server and check that unicorn was started by deployer (using ps or top)
verify that the pidfile exists in APP_ROOT/tmp/pids/unicorn.pid and contains the current pid of the unicorn master
become deployer and try to send the HUP signal to unicorn (sudo su deployer and then kill -HUP PID_OF_UNICORN_MASTER)
if this works, try /etc/init.d/unicorn_xxx restart as deployer
if this works but deploys with capistrano still won't work, please report back ;)

Update:

I quickly scanned your deploy.rb again and found something that might not be correct: please make sure that the directory APP_ROOT/tmp/pids is shared between all releases. Otherwise, unicorn writes its pidfile into this subdir in one release, and when you deploy a new release, unicorn can't find the pidfile since it is in a subdir in the old release... If you are using capistrano version 3, you can achieve this by including a line like

set :linked_dirs, [... 'tmp/pids' ...]

in your deploy.rb...

Another update (tl;dr):

To properly restart during a deploy, unicorn needs a valid pidfile at the location specified in the config file. In addition, the user as which unicorn is run needs proper permissions to write to this pidfile.

It has become standard for Rails apps to store pidfiles in the APP_ROOT/tmp/pids directory.

Since capistrano creates a new (app) directory for each deploy, it needs some trickery to keep shared data accessible from all releases/deploys. It does so by linking a couple of directories (including tmp/pids) to directories in a shared location.

If all this is happening correctly, then restarting unicorn will work correctly.

(I will update this answer as soon as the problem is completely resolved. Please also read the discussion in the comments to this answer...)

Unicorn with Capistrano restart doesn't work

4 Answers

Update:

Another update (tl;dr):