AWS Load Balancer EC2 health check request timed out failure

12
votes

I'm trying to get down and dirty with DevOps and I'm running into a health check request timed out failure. The problem is my Elastic Load Balancer sends a health check to my EC2 instance and gets a network timeout. I'm not sure what I did wrong. I am following this tutorial and I have completed all the steps up to and including "Using a Elastic Load Balancer". My EC2 instance seems to be working fine and I am able to successfully curl localhost on port 9292 from within the EC2 instance.

EC2 instance security group setup: description

Elastic Load Balancer setup: enter image description here

My target group for the ELB routing has port 9292 open via HTTP and here's a screenshot of the target in my target group that is unhealthy. enter image description here

Health check config:enter image description here

I have a VPC that my EC2 instance is a part of and my ELB is connected to the same VPC. I do not have Apache installed and I do not have nginx installed. To my understanding, I do not need these. I have a Rails Puma server running and I can send successful curl requests to the server.

My hunch is that my ELB is not allowed to reach my EC2 instance, resulting in a network timeout and a failed health check. I'm unable to find the cause for this. Any ideas? This SO post didn't help much. Are my security groups misconfigured? What else could potentially block a routing request from ELB to my EC2 instance?

Also, is there a way to view network requests / logs for my EC2 instance? I keep seeing VPC flow logging but I feel like there are simpler alternatives.

Here's something I posted in the AWS forums but to no avail.

UPDATE: I can curl the private IP of target just fine from within an EC2 instance. I don't think it's the target instance, I think it's something to do with the security group setup. I am unable to identify why though because I have basically allowed all traffic from the Load Balancer to the EC2 instance.

1
ruby-on-railsamazon-web-servicesamazon-ec2elastic-load-balancer
Can you show us the Health Check configuration on your load balancer? – John Rotenstein
added just now! – Sticky
I don't see any issues with your load balancer and VPC/security group configuration. I'm guessing your application simply isn't returning a 200 response code for the requests at /. Is your application perhaps returning a 301 response for that path? – Mark B
when I curl -I localhost:9292, it returns a 200 so I'm confused – Sticky
Are you able to curl -I $InstanceIP:9292 from another box in the subnet as the target, and member of the same security groups as the ALB? If yes, then make sure your ALB is in the same AZ as the target box. – Raf

1 Answers

8
votes

I made my mistake during the "Setup your VPC" step. I finished creating a subnet for an RDS instance. I proceeded to start an instance and the default subnet that AWS chose when I switched to my VPC was the subnet I made for my RDS, which was NOT a public subnet. Therefore, any attempts, from any EC2 instance or my load balancer, would not be able to reach it because I had only set up my public subnet to take requests.

The solution was to create a new instance and this time, pick the correct public subnet. My original EC2 instance was associated with a private subnet while the load balancer was pointing to the public subnet.

Here's a link to a hand drawn image that helped me pin point my problem, hopefully can help anyone else who's having trouble setting up. I didn't put image here directly because it's bigger than 2MB.

Glad to answer any further questions too!