Lambda cannot connect to RDS in VPC

Question

I have a VPC with RDS available in a private subnet. I can connect to this from an EC2 box from within the subnet. However, my Lambdas cannot connect!

Please could you look at the following configuration and spot my mistake?

Lambda config:

$ aws lambda get-function-configuration --function-name test
{
    "FunctionName": "test",
    "Role": "arn:aws:iam::xxxx:role/lambda_role",
    ...
    "VpcConfig": {
        "SubnetIds": [
            "subnet-00f3f0cb6957dbefa",
            "subnet-0d3d2cf4df53a862f"
        ],
        "SecurityGroupIds": [
            "sg-018da51b77f57eabf"
        ],
        "VpcId": "vpc-0704ca4d3f652fe9e"
    },
    ...
    "RevisionId": "e55b6fa2-998a-4b18-a620-69a218882b4e"
}

Execution role:

$ aws list-attached-role-policies --role-name lambda_role
{
    "AttachedPolicies": [
        {
            "PolicyName": "AWSLambdaVPCAccessExecutionRole",
            "PolicyArn": "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
        }
    ]
}

VPC:

$ aws ec2 describe-vpcs --vpc-ids vpc-0704ca4d3f652fe9e
{
    "Vpcs": [
        {
            "CidrBlock": "10.1.0.0/16",
            "DhcpOptionsId": "dopt-7764271f",
            "State": "available",
            "VpcId": "vpc-0704ca4d3f652fe9e",
            "InstanceTenancy": "default",
            "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-0c110a5fa85eb8883",
                    "CidrBlock": "10.1.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],
            "IsDefault": false,
            "Tags": []
        }
    ]
}

Security Group:

$ aws ec2 describe-security-groups --group-ids sg-018da51b77f57eabf
{
    "SecurityGroups": [
        {
            "Description": "Security group for Lambdas",
            "GroupName": "lambda-sg",
            "IpPermissions": [],
            "OwnerId": "xxxxx",
            "GroupId": "sg-018da51b77f57eabf",
            "IpPermissionsEgress": [
                {
                    "FromPort": 0,
                    "IpProtocol": "tcp",
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "Ipv6Ranges": [],
                    "PrefixListIds": [],
                    "ToPort": 65535,
                    "UserIdGroupPairs": []
                }
            ],
            "VpcId": "vpc-0704ca4d3f652fe9e"
        }
    ]
}

RDS security group (specifies both public and private subnets):

$ aws ec2 describe-security-groups --group-ids sg-0fbf7205b5d5fa98c
{
    "SecurityGroups": [
        {
            "Description": "Security group for RDS instance",
            "GroupName": "rds-sg",
            "IpPermissions": [
                {
                    "FromPort": 3306,
                    "IpProtocol": "tcp",
                    "IpRanges": [
                        {
                            "CidrIp": "10.1.2.0/24"
                        },
                        {
                            "CidrIp": "10.1.1.0/24"
                        },
                        {
                            "CidrIp": "10.1.4.0/24"
                        },
                        {
                            "CidrIp": "10.1.3.0/24"
                        }
                    ],
                    "Ipv6Ranges": [],
                    "PrefixListIds": [],
                    "ToPort": 3306,
                    "UserIdGroupPairs": []
                }
            ],
            "OwnerId": "xxxxxx",
            "GroupId": "sg-0fbf7205b5d5fa98c",
            "IpPermissionsEgress": [],
            "VpcId": "vpc-0704ca4d3f652fe9e"
        }
    ]
}

Linked: AWS Lambda Function not joining VPC

what are security groups defined on your rds instance? Ideally, you want to allow security groups on RDS instance to be accessible from security groups that are attached to Lambda. — toske
I added this to the answer, and I will test your suggestion. Do you happen to know if I can modify security groups on the fly for Lambda? Do I need to wait for the lambda to die and be recreated? — mafrosis
not sure what 'on the fly' means, but if you mean outside of cloudformation - yes it is possible, but not advisable. — toske

Eric des Courtis Eric des Courtis · Accepted Answer · 2018-10-20T20:45:49

Try to enable ICMP on any security group and any network ACL. It could simply be a PMTUD blackhole situation.

Lambda cannot connect to RDS in VPC

1 Answers