S3 copy inside lambda function timing out

Question

I've got a Lambda function written in Python using Boto3 and setup to fire off when a file is put into an S3 bucket. I want to copy that file to another S3 bucket across account. Here is the code:

source =  boto3.resource('s3')
destination = boto3.resource('s3',
    aws_access_key_id='<redacted>',
    aws_secret_access_key='<redacted>')
source_bucket = source.Bucket(bucket)
destination_bucket = destination.Bucket('targetBucket')
copy_source = {
    'Bucket': bucket,
    'Key': key
}
destination_bucket.copy(copy_source, key)

When the Lambda function fires, everything seems to go well, but the copy never happens. When that line fires off, I get the following message in my logs. It looks like the connection to the S3 bucket is timing out. I've verified that it doesn't appear to be a permissions issue.

[INFO] 2017-01-25T19:33:59.400Z c942ee38-e334-11e6-850d-45d8f93ed656 Starting new HTTPS connection (1): [source-bucket].s3.amazonaws.com

Would anyone have any ideas?

How big is the file? It's possible that it's just too big to copy in the time frame. — Jordon Phillips
The files are pretty small. I'm using jpeg files around 1 Mb for testing. Actual usage will be small xml files. — Xanxir
The function is in a VPC, but in trying to debug, I temporarily opened the bucket to the world. I was able to hit the file in S3 from a browser outside of AWS. But it's not something directly occurred to me. I'll try poking around that. — Xanxir
In a VPC, the Lambda function must be associated with a private subnet, equipped with a NAT device, or you need an S3 VPC Endpoint. — Michael - sqlbot

Xanxir Xanxir · Accepted Answer · 2017-01-28T05:09:03

1

votes

As mentioned by @Michael-sqlbot above, I needed an S3 VPC Endpoint.

S3 copy inside lambda function timing out

1 Answers