AWS Lambda in VPC sometimes doesn't have internet access

Question

I have Lambda which was deployed to VPC.

This deploymens has next configs:

VPC (192.168.0.0/16)
Public Subnet A (192.168.32.0/20) has NAT Gateway and Route 0.0.0.0/0 to Internet Gateway
Private Subnet A (192.168.48.0/20) has Route 0.0.0.0/0 to NAT Gateway
Private Subnet B (192.168.64.0/20)

Lambda has own Securiy Group and references to "Private Subnet A" and "Private Subnet B"

I have strange problem: time to time Lambda doesn't have Internet Access. 3rd party service works normal.

One more strange thing that Lambda gets IP's like 127.0.0.1, 169.254.76.13, 169.254.79.1 instead of IP's from Subnets (192.168.48.0/20 and 192.168.64.0/20).

Error:

Error: connect ETIMEDOUT x.x.x.x:443
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)

Deployment schema:

Here full CloudFormation template:

    ---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Base Infrastructure'
Metadata:
  'AWS::CloudFormation::Interface':
    ParameterGroups:
    - Label:
        default: 'VPC Parameters'
      Parameters:
      - VpcId
      - InternetGatewayId
Parameters:
  VpcId:
    Type: String
  InternetGatewayId:
    Type: String
Resources:
  SubnetAPublic:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [0, !GetAZs '']
      CidrBlock: !Sub '192.168.32.0/20'
      MapPublicIpOnLaunch: true
      VpcId: !Sub '${VpcId}'
  SubnetAPrivate:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [0, !GetAZs '']
      CidrBlock: !Sub '192.168.48.0/20'
      VpcId: !Sub '${VpcId}'
  SubnetBPrivate:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: !Select [1, !GetAZs '']
      CidrBlock: !Sub '192.168.64.0/20'
      VpcId: !Sub '${VpcId}'
  RouteTablePublic:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Sub '${VpcId}'
  RouteTablePrivate:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Sub '${VpcId}'
  RouteTableBPrivate:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      VpcId: !Sub '${VpcId}'
  RouteTableAssociationAPublic:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref SubnetAPublic
      RouteTableId: !Ref RouteTablePublic
  RouteTableAssociationAPrivate:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref SubnetAPrivate
      RouteTableId: !Ref RouteTablePrivate
  RouteTableAssociationBPrivate:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      SubnetId: !Ref SubnetBPrivate
      RouteTableId: !Ref RouteTableBPrivate
  EIP:
    Type: 'AWS::EC2::EIP'
    Properties:
      Domain: vpc
  NatGateway:
    Type: 'AWS::EC2::NatGateway'
    Properties:
      AllocationId: !GetAtt 'EIP.AllocationId'
      SubnetId: !Ref SubnetAPublic
  RouteTablePublicInternetRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref RouteTablePrivate
      DestinationCidrBlock: '0.0.0.0/0'
      NatGatewayId: !Ref NatGateway
  RouteTablePublicInternetRoute2:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref RouteTablePublic
      DestinationCidrBlock: '0.0.0.0/0'
      GatewayId: !Sub '${InternetGatewayId}'
  ServerlessSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SecurityGroup for Serverless Functions
      VpcId: !Sub '${VpcId}'
  ServerlessSecurityGroupIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref ServerlessSecurityGroup
      IpProtocol: -1
      SourceSecurityGroupId: !Ref ServerlessSecurityGroup

Any ideas what I do wrong?

P.S.: I found similar issues AWS VPC Lambda Function keeps losing internet access and Why AWS lambda functions In a VPC sometimes timeout and sometimes work fine?

UPD

Added Route and it works now

  RouteTableBPrivateInternetRoute:
    Type: AWS::EC2::Route
      Properties:
        RouteTableId: !Ref RouteTableBPrivate
        DestinationCidrBlock: '0.0.0.0/0'
        NatGatewayId: !Ref NatGateway

Mark B Mark B · Accepted Answer · 2018-07-17T12:39:52

For an AWS Lambda function running inside a VPC to be able to access resources outside the VPC (such as the Internet), it must be in a private subnet with a NAT gateway. In your instance Private Subnet A is the only subnet with the appropriate configuration to allow a Lambda function to access the Internet. So you need to edit your Lambda function's configuration to only run in that subnet.

AWS Lambda in VPC sometimes doesn't have internet access

2 Answers