Export DynamoDB Table as JSON to S3 Using Lambda Function

Question

I am trying to export DynamoDB table as JSON format to S3 and from there import it to BigQuery. The important part is exporting DynamoDB table as JSON format to S3, because the table I am working on is not a small table. This table contains 5.6 million records and about 15.000 (on a quiet day) new records are inserted every day. I came across with a blog post which suggests Lambda (ref: http://randomwits.com/blog/export-dynamodb-s3) function but table.scan() function does not work well with large tables.

So how can I export DynamoDB table in JSON format to S3 and from there import it to BigQuery efficiently? I saw some options like HEVO, Glue, etc. But I don't know which way would be the most efficient.

DynamoDB has a new feature "export to S3" which offers a good solution but in that case I will have to enable Point-in-time recovery (PITR) for Amazon DynamoDB. I am unsure if it will worth it, if the solution will be effective. — M. Ece Ercan
I would enable PITR, personally. Related: stackoverflow.com/questions/18896329/export-data-from-dynamodb and stackoverflow.com/questions/33357821/…. — jarmod
@M.EceErcan Please go through the below link. aws.amazon.com/blogs/aws/… — Sarathy Velmurugan

Levi Levi · Accepted Answer · 2020-11-18T11:14:35

You can do this with AWS lambda, lambda is triggered by DynamoDB stream, then this lambda will write to cloud logging, from cloud logging you will have to create a sink and make big query as the destination

Export DynamoDB Table as JSON to S3 Using Lambda Function

1 Answers