Exporting data over an api from s3 using lambda

Question

I have some data stored in dynamo db and some highres images of each user stored in S3. The requirement is to be able to export a users data on demand. So by an api endpoint, collate all data and send it as a response. We are using aws lambda using node.js for business logic, s3 for storing images and sql db for storing relational data

I had set up a mechanism to connect api gateway to receive requests and put them in a sqs. The sqs would trigger a lambda which would run queries to gather all the data and image paths. We would copy all the images and data into a new bucket with custId as a folder name. Now heres where Im stuck. How to stream this data from our new aws bucket. All collected data is about 4gb. I have tried to stream via aws-lambda but keep failing. I am able to stream sigle files but not all data as zip. Hv done this in node, but would not want to set up an EC2 is possible and try to solve it directly with s3 and lambdas

CAnt seem to find a way to stream an entire folder from aws to the client as a response to an http request

what do you mean by stream? why not just create a signed url and send it to clients and let them take care of downloading? — best wishes
i tried that, but was able to create signed urls for single files only not the whole folder. By stream i mean the whole file is not loaded in memory but piped as its read — noob7
It would be advisable to zip the files so the customer only needs to download one file. You could then use a pre-signed URL. — John Rotenstein
But im not entirely sure how to zip them without downloading. The only solution i could think of was download all contents in an EC2, zip and stream them. But thats the last resort, I really hope theres a solution with lambda. Please note that the images we need are already on a different lambda so we are just copying them in this lambda instead of uploading them again in which case we would hv had a point where we could zip them — noob7
on second tht im not sure even ec2 would work☹️ we are talking multiple files upto 4gb per user. any help advise or general direction would be welcome — noob7

noob7 noob7 · Accepted Answer · 2019-06-23T13:58:18

Okay found the answer. Instead of trying to return a zip stream, Im now just zipping and saving the folder on the bucket itself and returning a signed url for it. Many node modules help us zip s3 folders without loading entire files in memory. Using that we have zipped our folder and returned a signed url. How it will behave under actual load remains to be seen. Will do that soon

Exporting data over an api from s3 using lambda

1 Answers