My ROR+Nginx+Passenger+AWS is sending 502 bad gateway

0
votes

I am developing android service with ROR+Nginx+passenger and Amazon Web service.

The server goes well, but yesterday the server suddenly got downed.

I've tried to solve the problem, but I couldn't.

Here's my problem.

First, the EC2 instance's log/production.log doesn't write any log. I can get 502 bad gateway error message in my android app log.

Second, all requests including http, https are responded by 502 badgateway.

Third, my AWS load balancer is logging 50X error.

I guess this problem is about ELB, but I don't know how to solve.

I have ssl certificate, and I have only one EC2 instance.

Any help will be appreciated.

Edited

2017-02-22 00:22:30.9139 25313/7f222067b700 age/Cor/Con/CheckoutSession.cpp:269 ]: [Client 1-356202] Returning HTTP 503 due to: Request queue full (configured max. size: 100)
[ 2017-02-22 00:24:00.3704 25313/7f221ee09700 age/Cor/App/Implementation.cpp:304 ]: Could not spawn process for application /home/ec2-user/popcake: An error occurred while starting up the preloader: it did not write a startup response in time.
  Error ID: 10045ede
  Error details saved to: /tmp/passenger-error-JS1Bnt.html
  Message from application: An error occurred while starting up the 

preloader: it did not write a startup response in time. Please read <a href="https://github.com/phusion/passenger/wiki/Debugging-application-startup-problems">this article</a> for more information about this problem.<br>
    <h2>Raw process output:</h2>
    (empty)

    [ 2017-02-22 00:24:00.4527 25313/7f222067b700 age/Cor/Con/CheckoutSession.cpp:285 ]: [Client 1-353433] Cannot checkout session because a spawning error occurred. The identifier of the error is 10045ede. Please see earlier logs for details about the error.
[ 2017-02-22 00:30:29.0872 25313/7f222067b700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2017-02-22 00:30:29.1173 25313/7f2227bfc840 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2017-02-22 00:30:29.1174 25313/7f221fe7a700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
[ 2017-02-22 00:30:29.1174 25313/7f221fe7a700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished

[ 2017-02-22 00:30:29.2400 25318/7f77237e2700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2017-02-22 00:30:29.2741 25318/7f772a775840 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2017-02-22 00:30:29.3043 25318/7f77237e2700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
[ 2017-02-22 00:30:29.3042 25318/7f7722fe1700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
[ 2017-02-22 00:30:29.3048 25318/7f7722fe1700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
[ 2017-02-22 00:30:29.3720 25318/7f772a775840 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished

[ 2017-02-22 00:30:29.4023 25313/7f222067b700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 1 more time(s) to force shutdown)
1
ruby-on-railsamazon-web-servicesamazon-ec2
check the Nginx error log file to see error. – rails_id
@rails_id how can I access to Nginx error file? – Jeongmyeong Lee
@JeongmyeongLee you can access this path: /var/log/nginx/ and find error file. It same as tail -f /var/log/nginx/error.log – Khanh Pham
I eddited some featured errors. Almost of logs are same as above. – Jeongmyeong Lee
and I found this log at /opt/nginx/logs/error.log. It depends on version of nginx or something. – Jeongmyeong Lee

1 Answers

0
votes

If you see the log that you have added it says "Request queue full", basically you have some api's which are long running, because of which 50x is being thrown after the queue was full.

Restarting nginx might fix the issue for now but you need to look at the response times of your apis and see which are slow performing and fix them otherwise it will be a recurring issue and you will have to restart nginx everyday.