How to download and upload s3 objects in same bucket (different prefixes) with boto3

Question

I'm implementing an image processing algorithm with an AWS lambda function. I want to gather images from a 'subfolder'/prefix in an S3 bucket, run my algorithm with boto3, and upload the processed images to a different 'subfolder'/prefix in the same S3 bucket.

I have not yet been successful with moving the images around within the same S3 bucket or under prefixes. I am able to use a boto3 resource or client to download an image from the 'root' folder in the bucket and then upload the processed image to a different bucket. However, I have been unsuccessful in accessing an image that is in a 'subfolder'/prefix.

def lambda_handler(event, context):
    for record in event['Records']:
    bucket = record['s3']['bucket']['name']
    key = record['s3']['object']['key'] 
    file_path = '/tmp/' + key

    s3_client.download_file(bucket, key, file_path)

    # call image processing algorithm here

    s3_client.upload_file(file_path, 'bucket/folder_a/folder_b', key)

The key that is being passed in with the event is 'folder_a/folder_b/IMG_X.jpg'. I've been getting file not found errors.

Answer 1

It looks like you're putting the path as part of the bucket name. Add it instead to the beginning of the key like:

s3_client.upload_file(file_path, 'bucket', 'folder_a/folder_b' + key)

Answer 2

I was getting file not found errors because boto3 didn't like the way I was passing the key to the download_file function. Instead of passing the key with the entire folder structure (as the S3 console shows the key to be), I just passed the base file name.

For the upload_file function it is necessary to pass the entire folder structure (with the file name) of where you want the file to be stored in the S3 bucket.

s3_client = boto3.client('s3')

def lambda_handler(event, context):

    for record in event['Records']:
        try:
            bucket = record['s3']['bucket']['name']
            key = record['s3']['object']['key']
            file_name = key.split('/')[-1]
            file_path = '/tmp/' + file_name

            s3_client.download_file(bucket, key, file_path)

            # call image processing algorithm here

            s3_client.upload_file(file_path, <target bucket>, 'folder_a/folder_b' + file_name)

            return {
                'statusCode': 200,
                'body': 'Success'
            }
        except Exception as e:
            logger.info(e)
            return {
                'statusCode': 400,
                'body': 'Error'
            }