502 Bad Gateway - Nginx - Unicorn suddenly stopped working, Nginx logs empty

Question

BIG EDIT WITH THE ANSWER

If your Nginx application throws you a "502 Bad Gateway", check inside your Nginx conf files /etc/nginx/conf.d/*, probably you might have the same domain declared twice in different .conf files. Nginx error logs WON'T show you anything.

=====

I have a few Rails Applications running on same Centos VPS. Those apps are running with Nginx and Unicorn. Today I noticed that one app throws a "502 Bad Gateway" error. Other Rails Application are fine.

Therefore, I'm not sure what might have caused this error, but yesterday I did some changes to VPS, I changed my ssh port and then executed shutdown. Other Apps are fine, only that one isn't. However, I'm not sure if that is the cause.

I've been trying restarting both Unicorn and Nginx, but website still doesn't respond. What is really frustrating is that /var/log/nginx error logs is empty, Rails production log is also empty. Only Unicorn stderr log has some kind of info:

myweb/current/log/unicorn.stderr.log

INFO -- : worker=1 ready
INFO -- : reaped #<Process::Status: pid 595 exit 0> worker=0
INFO -- : reaped #<Process::Status: pid 598 exit 0> worker=1
INFO -- : master complete
INFO -- : Refreshing Gem list
INFO -- : unlinking existing socket=/var/sockets/unicorn.myweb_production.sock
INFO -- : listening on addr=/var/sockets/unicorn.myweb_production.sock fd=11
INFO -- : worker=0 ready
INFO -- : master process ready
INFO -- : worker=1 ready

I don't think there is a problem with my Nginx.conf file, It's template-generated and it works with all webs. Here it goes:

/etc/nginx/conf.d/miweb.conf

upstream unicorn_myweb_production {
  server unix:/var/sockets/unicorn.myweb_production.sock fail_timeout=0;
}
server {
  listen 80;
  client_max_body_size 4G;
  keepalive_timeout 10;

  error_page 500 502 504 /500.html;
  error_page 503 @503;

  server_name myweb.com www.myweb.com;
  root /var/www/myweb/current/public;
  try_files $uri/index.html $uri @unicorn_myweb_production;

  location @unicorn_myweb_production {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;

    proxy_pass http://unicorn_myweb_production;
    # limit_req zone=one;
    access_log /var/log/nginx/myweb_production.access.log;
    error_log /var/log/nginx/myweb_production.error.log;
  }

  location = /50x.html {
    root html;
  }

  location = /404.html {
    root html;
  }

  location @503 {
    error_page 405 = /system/maintenance.html;
    if (-f $document_root/system/maintenance.html) {
      rewrite ^(.*)$ /system/maintenance.html break;
    }
    rewrite ^(.*)$ /503.html break;
  }

  if ($request_method !~ ^(GET|HEAD|PUT|POST|DELETE|OPTIONS)$ ){
    return 405;
  }

  if (-f $document_root/system/maintenance.html) {
    return 503;
  }

  location ~ \.(php|html)$ {
    return 405;
  }
}

Unicorn file:

myweb/shared/config/unicorn.rb

working_directory "/var/www/myweb/current"
pid "/var/www/myweb/shared/tmp/pids/unicorn.pid"
stdout_path "/var/www/myweb/shared/log/unicorn.stdout.log"
stderr_path "/var/www/myweb/shared/log/unicorn.stderr.log"

listen "/var/sockets/unicorn.myweb_production.sock"

worker_processes 2
timeout 30

preload_app true

before_exec do |server|
  ENV["BUNDLE_GEMFILE"] = "/var/www/myweb/current/Gemfile"
end

before_fork do |server, worker|
  # Disconnect since the database connection will not carry over
  if defined? ActiveRecord::Base
    ActiveRecord::Base.connection.disconnect!
  end

  # Quit the old unicorn process
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end

  if defined?(Resque)
    Resque.redis.quit
  end

  sleep 1
end

after_fork do |server, worker|
  # Start up the database connection again in the worker
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end

  if defined?(Resque)
    Resque.redis = 'localhost:6379'
  end
end

Your rails app and unicorn are definitely working, must be problem from the Ngnix side. What kind off error you getting in the browser? Also try putting some dummy index.html in /var/www/myweb/current/public and try to restart Nginx, then open browser, just of purpose of pinpoiting the problem, if index.html is not shown then problem is in webserver and not in rail. — Vladan Markovic
The error is 502 Bad Gateway. I have tried adding index.html inside /public and restarting nginx, but it's the same. Other websites witn Nginx in same VPS work fine. — Ricardo Castañeda
Just for testing, inside deploy/production.rb I added two new domains as nginx_server_name, with those the application works. But it doesn't with the former domain. Any hint? — Ricardo Castañeda

Ricardo Castañeda Ricardo Castañeda · Accepted Answer · 2015-06-03T21:34:45

After many hours, I found that the source of my problem was that (by mistake) I had two different /etc/nginx/conf.d/ files using the same server_name. That was the source of my 502 Bad Gateway.

502 Bad Gateway - Nginx - Unicorn suddenly stopped working, Nginx logs empty

1 Answers