AWS VPC: Strange Behaviour when using NAT and Internet Gateway with Load Balancer and Private Subnets

Question

Consider the following setup based on this document:

• An AWS VPC with four subnets. One public and three private (one for each availability zone)
• The VPC has an Internet Gateway attached to it.
• The public subnet (10.0.1.0/24) has an Elastic Load Balancer (V2), a NAT Gateway and a Bastion server for SSH'in into the environment. The routing table for this subnet is defined as: 10.0.0.0/16 -> local 0.0.0.0/0 -> igw-67e14203 (Internet Gateway)
• The three private subnets (on in each Availability Zone) have the following routing table attached: 10.0.0.0/16 -> local 0.0.0.0/0 -> igw-67e14203

With the above setup, the load balancer works perfectly and I can reach the web server urls and applications from the public internet. However, with this setup, the servers in the private subnet (10.0.2.0/24,10.0.3.0/24,10.0.4.0/24) are not able to access anything outside the local network - not even the AWS yum repositories. When I change the routing table for the private subnets to: 10.0.0.0/16 -> local 0.0.0.0/0 -> nat-0a71345c417d7758a

• If I look at the Health Checks under Target Groups, it shows all instances in all three private subnets as healthy.
• Unless I am missing something, as the per the document referenced above, the load balancer can, in fact, be connected to the private subnet(s).

The configuration for ELB is as follows:

"AppServerLoadBalancer": {
      "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer",
      "Properties": {
        "Scheme": "internet-facing",
        "Tags": [
          {
            "Key": "environment",
            "Value": {
              "Ref": "Environment"
            }
          }
        ],
        "SecurityGroups": [
          {
            "Ref": "LoadBalancerSecurityGroup"
          }
        ],
        "Subnets": [
          {
            "Ref": "AppServerSubnetAZ0"
          },
          {
            "Ref": "AppServerSubnetAZ1"
          },
          {
            "Ref": "AppServerSubnetAZ2"
          }
        ]
      }
    }

The subnets AppServerSubnetAZ0, AppServerSubnetAZ1 and AppServerSubnetAZ2 are private subnets with a route that points to NAT Gateway as described earlier.

The instances in the private subnet can access outside Internet but then the LoadBalancer stops working. I start getting timeouts on the load balancer.

The Network ACL's are set properly and the only change in the two above scenarios is the change in the routing table.

Just can't seem to figure out what is going wrong? I would have assumed that the NAT gateway would have taken care of routing load balancer traffic as well as in the article/link above?

Thank you for your help!

Answer 1

I don't know how feasible it is to make both a NAT gateway and ELB to work for EC2 instances in Private subnet.

One work around I will suggest you is keep your Routing table

10.0.0.0/16 -> local
0.0.0.0/0 -> igw-67e14203 (Internet Gateway)

And achieve the NAT using an EC2 instance instead of NAT Gateway.

Answer 2

You are misinterpreting the referenced document.

The load balancer must be on a public subnet, and the instances should be on a private subnet.

Why does this work when the route on the private subnet points to the Internet Gateway? That's a trick question. If the defaut route points to the Internet Gateway, it's not actually a private subnet any more. It's a public subnet.

Don't think in terms of conventional networks, where it makes sense that the balancer and the instances behind it would often be on a common subnet. VPC doesn't work that way. There's no performance penalty for traffic across subnet boundaries within an availability zone, nor a performance advantage for traffic within a single subnet.