COPY csv data into Redshift using LAMBDA

Question

I'd like to create a Lambda function that COPY on Redshift the data of a file PUT in a specific S3 Bucket, but I can't figure how to do that.

So far, I created a LAMBDA function that triggers whenever a .csv file is PUT on the S3 bucket and I managed to COPY the data from a local .csv file to Redshift.

Now I'd like to be helped on how to COPY the data using a Lambda function. I searched on the internet, but can't manage to have proper examples using Lambda.

Amazon provides a work solution for this: AWS Lambda Redshift Loader. — Jonathan
@Jonathan I've started to work with that already, should have mentionned it in the original post. I was looking for a "simpler" solution, since I will copy like 2 or 3 files per week, I do not need such a huge processing system for that. But thank you ! I am currently close to the solution using AWS Lambda Redshift Loader, still having some issues I can't deal with, but I'll manage to. — Prototype
I understand, hopefully you can just take the parts you need from the example. Good luck. — Jonathan
Welp, managed to do what I wanted using another method : g14n.info/2016/04/s3-to-redshift-loader It works fantastically great and managed to deal with a lot of files ( I tried with 200 files at once, and every data has been processed. ) EDIT : Should I put this as an answear of my own question in order to help ? — Prototype

CrazyNew CrazyNew · Accepted Answer · 2019-06-28T08:34:05

I use powershell to export data from SQL Server and upload it to an S3 bucket. I then have a Lambda function with an S3 put trigger that executes a stored procedure in Redshift that contains the copy command in a dynamic statement, in order to load data into different tables, and it will trigger's every time a file is uploaded, so multiple tables multiple times.

COPY csv data into Redshift using LAMBDA

2 Answers