AWS Application Load Balancer sends request to unhealthy target groups / instances

Question

In my AWS VPC I have an application load balancer and two target groups. Each target group has only 1 instances in different az with health check setup.

Now when one of my instance in a target group goes down AWS correctly marks its as unhealthy. But my application load balancer still keeps sending requests to it, causing my application to work intermittently.

But in the documentation it's clearly mentioned that unhealthy instances are ignored by the load balancer.

I checked all the LB and TG configuration none suggest that they shouldn't be ignored.

It monitors the health of its registered targets, and routes traffic only to the healthy targets.

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html

Answer 1

There is an exception from this rule. From docs:

If a target group contains only unhealthy registered targets, the load balancer nodes route requests across its unhealthy targets.

So basically, if all targets are unhealhty, ALB will ignore this fact and attempt to route requests to the targets anyway.