Unable to connect to S3 from Lambda/Python/Boto3 when VPC is enabled

Question

I have a very simple python function in a lambda which runs fine if I leave VPC disabled.

import json
import boto3
import botocore

    def lambda_handler(event, context):

    s3 = boto3.client('s3', 'us-east-1', config=botocore.config.Config(s3={'addressing_style':'path'}))
    keys = []
    resp = s3.list_objects_v2(Bucket='[BUCKET_NAME]')
    for obj in resp['Contents']:
        print(obj['Key'])

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

When VPC is enabled the S3 connection continually times out.

I have gone through many documents, tutorials, forum threads and stack overflow postings, but none of them have helped me.

My network ACL has 0.0.0.0/0 mappings for ports 80, 443 and 5439 (Redshift).

My one and only security group has 0.0.0.0/0 mappings for ports 80, 443 and 5439 (Redshift).

I have only one VPC configured.

I have 1 NAT Gateway configured.

I have 1 Internet Gateway configured.

I have 6 subnets in the VPC:

Subnets A and B point to the main route table.
Subnets C and D point to the 'lambda_rt_table_gateway' route table.
Subnets E and F point to the 'lambda_rt_table_nat' route table.

I have 2 endpoints in the VPC:

Endpoint VPCE-A is defined for service 'com.amazonaws.us-east-1.s3' and is mapped to all 3 route tables.
Endpoint VPCE-B is defined for service 'com.amazonaws.us-east-1.dynamodb' and is mapped to all 3 route tables.

Finally, I have 3 Route Tables:

The main route table has the following routes:
- 172.31.0.0/1 --> local
- pl-02cd2c6b (com.amazonaws.us-east-1.dynamodb, 52.94.0.0/22, 52.119.224.0/20) --> vpce-07a6eb423bbbea151
- pl-63a5400a (com.amazonaws.us-east-1.s3, 54.231.0.0/17, 52.216.0.0/15) --> vpce-0fd10c890bb176b5a
- 0.0.0.0/0 --> igw-04b6aa7c
The 'lambda_rt_table_gateway' route table has identical routes as the main.
The 'lambda_rt_table_nat' route table has identical routes as well except for the last entry, it is
- 0.0.0.0/0 --> nat-0a5c0a76e3c12c42f

I am pretty sure it is something simple I'm missing. Please help.

Thanks a lot.

Question: How much of the above have you created to specifically solve this situation vs having it for other purposes? For example, is something else using the NAT Gateway or VPC Endpoint? — John Rotenstein
All this configuration was done attempting to solve this issue. It can all be removed if that is the right approach. This is my test environment and I want to figure out the right way to solve this problem. — Garet Jax
I am using the addressing_style:path because of this SO thread: stackoverflow.com/questions/39779962/… — Garet Jax
Probably the worst possible provisioning error I see people make is changing Network ACLs before ensuring that the setup otherwise works as expected. I see no allowance mentioned for ephemeral ports. Set the Network ACLs back to normal (allowing all in both directions) before proceeding. — Michael - sqlbot

John Rotenstein John Rotenstein · Accepted Answer · 2019-02-06T03:56:54

You have a lot of stuff configured! I'm not sure how much of it is part of wanting to get this specific situation fixed, or whether you have other needs for things like the NAT Gateway, VPC Endpoints, etc.

The simplest setup to enable a VPC-connected Lambda function to call out to the Internet (eg to make an API call to Amazon S3) would be:

Add a NAT Gateway to a Public subnet
Attach the Lambda function to a Private subnet
Set routing on the private subnet to use the NAT Gateway for 0.0.0.0/0

That is sufficient for VPC-attached Lambda functions to reach the Internet.

Unable to connect to S3 from Lambda/Python/Boto3 when VPC is enabled

1 Answers