Using S3 upload to trigger Lambda function to create EBS volume, which would create EBS volume based on S3 upload size (using boto3)

Question

Currently, I am using a S3 trigger to launch a lambda function once a file is placed in S3.

My intent is to get the size from the S3 upload and pass that value into the Size field in an automated function that creates an EBS volume.

I am attempting to then create an EBS volume based upon the file size. Then copy the file to the EBS volume that will be attached to an EC2 instance that will then process the file. I am using EBS volumes because the files that get uploaded to the main S3 bucket are compressed files that need to be uncompressed and processed.

Is there a way to create an EBS volume Size based upon the original file that was uploaded?

response = client.create_volume(
    AvailabilityZone='string',
    Encrypted=True|False,
    Iops=123,
    KmsKeyId='string',
    OutpostArn='string',
    Size=123,
    SnapshotId='string',
    VolumeType='standard'|'io1'|'gp2'|'sc1'|'st1',
    DryRun=True|False,
    TagSpecifications=[
        {
            'ResourceType': 'client-vpn-endpoint'|'customer-gateway'|'dedicated-host'|'dhcp-options'|'elastic-ip'|'fleet'|'fpga-image'|'host-reservation'|'image'|'instance'|'internet-gateway'|'key-pair'|'launch-template'|'natgateway'|'network-acl'|'network-interface'|'placement-group'|'reserved-instances'|'route-table'|'security-group'|'snapshot'|'spot-fleet-request'|'spot-instances-request'|'subnet'|'traffic-mirror-filter'|'traffic-mirror-session'|'traffic-mirror-target'|'transit-gateway'|'transit-gateway-attachment'|'transit-gateway-multicast-domain'|'transit-gateway-route-table'|'volume'|'vpc'|'vpc-peering-connection'|'vpn-connection'|'vpn-gateway'|'vpc-flow-log',
            'Tags': [
                {
                    'Key': 'string',
                    'Value': 'string'
                },
            ]
        },
    ],
    MultiAttachEnabled=True|False
)

It seems most strange that you wish to create an Amazon EBS Volume based on the size of one file. Could you possibly explain your wider use-case for wanting to do this? I'm also confused that your question starts by referencing an upload to Amazon S3 and (I presume) wanting to get the size of that upload, but then swings to asking about Amazon EBS (which is not used by AWS Lambda functions). Could you possibly edit your question to clarify what you are actually seeking to accomplish? We will then be able to better assist you. — John Rotenstein
This sounds like an XY Problem (meta.stackexchange.com/questions/66377/what-is-the-xy-problem) — jarmod
There would be no way for the Lambda function to copy the file to the EBS volume, so I think you need to rethink this whole thing. I would recommend having an EC2 instance with a decent amount of EBS space that reads from an SQS queue for files it needs to process. S3 could send messages to the queue when new files arrive. Lambda doesn't seem to fit here at all if you are doing the processing on EC2. — Mark B

John Rotenstein John Rotenstein · Accepted Answer · 2020-03-24T01:07:24

It appears that your goal is to have an Amazon EC2 instance process a file that has been uploaded to Amazon S3.

Firstly, there is no need to send the file via an Amazon EBS volume. Instead, the software running on the EC2 instance can simply retrieve is from S3.

The main architectural decision is how to trigger the process on the EC2 instance when a file has been uploaded.

I would recommend this architecture:

Create an Amazon SQS Queue
Configure an Amazon S3 Event to push a message into the SQS queue when a file has been uploaded (this will include the bucket name and file name)
Have a process running on the Amazon EC2 instance that regularly checks the SQS queue (eg every minute, or more/less often depending upon your requirements)
If a message is found, the app should:
- Download the file from Amazon S3
- Process the file
- Delete the local file
- Optional: Delete the file from S3

Alternatively, you could process the file in AWS Lambda without using an Amazon EC2 instance, if possible. However, you did not provide enough information in your question to determine whether this is feasible.

Using S3 upload to trigger Lambda function to create EBS volume, which would create EBS volume based on S3 upload size (using boto3)

1 Answers