AWS Elastic Beanstalk health check issue

10
votes

My web application is Django and web server use Nginx, use Docker image and Elastic Beanstalk for deployment.

Normally there was no problem, but as the load balancer expands EC2, my web server becomes 502 Bad Gateway.

I checked Elastic Beanstalk application logs, about 16% of the requests returned 5xx errors, at which time the load balancer expands EC2, causing the web server to transition to the 502 Bad Gateway state and the Elastic Beanstalk application to the Degraded state.

Is this a common problem when the load balancer performs a health check? If not, how to turn off the Health Check?

I am attaching a captured image for reference.

enter image description here

2
pythondjangoamazon-web-servicesdeploymentamazon-elastic-beanstalk
Thank you for editing my post. – Seung
did you manage to find a fix? – George2456
Do you have application monitoring in place? Do you know what errors are actually being thrown? – Thomas
@Seung Is it only for the time the Instance just started? ( After some time, does the health checks return OK? ) – qkhanhpro
Load balance doesn't expand, Auto Scaling service expands. LB only removes bad instance. Can you please clarify problem and in which situation you see it – forvaidya

2 Answers

2
votes

As far as I know, 502 Bad Gateway error can be mitigated only by manually checking the major links you have on your websites and if they are accessible through a simple GET request.

In case of my website, I had some issue with the login page and an about page, (and it was about 33% of my website sadly) which is why after uploading to EC2 i got a 5xx error on health check. I solved the problem by simply making the links work on the server (there were some functionalities which were only running on localhost and not on AWS so I fixed that and got OK status in Health Check)

I don't think there is a point in removing health check as it gives vital information about your website and you probably don't want your website to have inaccessible pages.

Keep track of logs to narrow down to the problem.

I hope you find the solution.

1
votes

While your code is being deployed, you will get 502 because the EC2 instance fails the health check call. You need to adjust the load balance health check default settings to allow enough time for your deployment to complete. Allow more time for a deployment if you also restart the server after each deployment.

The AWS load balancer sends a health check request to each registered instance every N seconds using the path you specify. The default interval seconds is 30 seconds. If the health check fails N number of times (default is 2) for any of the instances you have running, health changes to Degraded or Severe depending on the percentage of your instances that are not responding.

  1. Send a request that should return a 200 response code. Default is '/index.html'
  2. Wait for N seconds before time out (default 5 seconds)
  3. Try again after N interval seconds (default 30 seconds)
  4. If N consecutive calls fail, change the health state to warning or severe (default unhealthy threshold is 2)
  5. After N consecutive successful calls, return the health state to OK (default is 10).

With the default settings, if any web server instance is down for more than a minute (2 tries of 30 seconds each), it is considered an outage. It will take 5 minutes (10 tries every 30 seconds) to get back to Ok status.

For a detailed explanation and configuration options please check AWS documentation: Configure Health Checks for Elastic Load Balancing