Can not access S3 via VPC endpoint in Lambda

Question

I have a Lambda function in my VPC, and I want to access S3 bucket.

I have set S3 VPC endpoint correctly I think,

because I created an EC2 instance in the same subnet(Lambda function subnet),

use the same security group,

and run the copy of Lambda function code,

It can correctly show the S3 file content.

But when I run the code in Lambda, it failed.

So, I want to know what is the difference between "run in EC2" and "run in Lambda"?

Why it failed when I run it in Lambda?

Here is my Lambda function code:

import boto3

s3 = boto3.client('s3', region_name='ap-northeast-1')

def lambda_handler(event, context):
    bucket = '*xxxxxx*'
    key = 's3-upload.json'
    try:
        response = s3.get_object(Bucket=bucket, Key=key)
        print('--------------------------------------')
        print(response)
        print('--------------------------------------')
        body = response['Body'].read()
        print(body)
        print('--------------------------------------')
        print("CONTENT TYPE: " + response['ContentType'])

    except Exception as e:
        print('Error getting object.')
        print(e)
        raise e

Answer 1

If you want to allow an AWS Lambda to access Amazon S3, use one of these methods:

• Do not associate the function to a VPC. Access is then automatic.
• If the function is attached to a public subnet in the VPC, associate an Elastic IP to the Lambda function's ENI that appears in the VPC (Not recommended)
• If the function is attached to a private subnet in the VPC, launch a NAT Gateway in the public subnet and update Route Tables. Traffic will flow to the Internet via the NAT Gateway.
• Add an Amazon S3 VPC Endpoint in the VPC and update Route Tables. Traffic will flow through that instead of the Internet Gateway.

Answer 2

Even though they're in the same VPC, EC2 and Lambda are still different environments within AWS. Being able to run your code in one and not the other implies that your code is fine and works, so it's likely to be a configuration issue with AWS.

Have you checked the service/execution role that the lambda is using?

You need to ensure that the IAM role that it's using is allowed the correct level of S3 access.

This documentation on execution roles for lambda might provide a useful jumping off point: https://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html#lambda-intro-execution-role

An IAM policy like this would give whatever execution role you use read-only access to all your S3 buckets, and happens to be one of the AWS managed policies.

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "s3:Get*",
            "s3:List*"
        ],
        "Resource": "*"
    }
]

}

Answer 3

Thanks everyone! I found the reason.

My Lambda have two subnets, private_sn_1 and private_sn_2,

private_sn_1 have correctly set the vpc endpoint route table,

but the private_sn_2 set a wrong route table,

and my ec2 created in private_sn_1 so it can access the vpc endpoint.

In normal, Lambda will run randomly in private_sn_1 or private_sn_2,

but in my case it always run in private_sn_2(I don't know why),

so when I fixed the private_sn_2 route table,

everything is right.

Answer 4

In addition to all said above, it is also possible that VPC Endpoint policy can be prohibitive and not allowing traffic to/from S3 through. Make sure you allow traffic through endpoint by using "Full access" policy.

Edit: here's related bit of documentation:

Your policy must contain a Principal element. For gateway endpoints only, you cannot limit the principal to a specific IAM role or user. Specify "*" to grant access to all IAM roles and users. Additionally, for gateway endpoints only, if you specify the principal in the format "AWS":"AWS-account-ID" or "AWS":"arn:aws:iam::AWS-account-ID:root", access is granted to the AWS account root user only, and not all IAM users and roles for the account.

So for S3 endpoints to work you need to specify '*' as a principal in general case