Copy DynamoDB table to another account

3
votes

How Do i Copy DynamoDB table to another AWS account ?

The table size is 20 GB.

I couldn't make the data pipe to work - I get WAITING_FOR_RUNNER forever, so I prefer a different approach

1
amazon-web-servicesamazon-dynamodb
stackoverflow.com/questions/43607592/… – raevilman

1 Answers

5
votes

The recommended and best-practice by AWS to do this is to use AWS Data Pipeline service while utilizing pre-defined pipeline templates named:

  1. Import DynamoDB backup data from S3
  2. Export DynamoDB table to S3

The full tutorial on this topic is well written and available in AWS Data Pipeline Docs.

The only two gotchas that tutorial doesn't cover are:

  1. That you will probably need to share S3 bucket between two AWS accounts using ACLs or bucket policies
  2. That you will have to change your DDB table from either "On Demand" or "Auto Scaling On" throughput provisioning modes to manual throughput and set your table with enough read/write capacity so that your EMR job can scale reasonably. Have in mind that ratio of RCUs and WCUs that EMR job will consume to perform data export/import operations is parameter that you set when creating data pipeline from above templates, and it can be changed later (if you edit pipeline in Architect). This planning ahead ensures that your EMR job doesn't get stuck trying to read data from DDB forever if your RCUs are for example set very low by Auto-Scaling based on slower usage pattern before you run export. And similar observation goes for WCUs when importing.

I can confirm migrating tables much larger than 20GB using this method and it was most reliable and fault-tolerant method for this case. Other methods based on some libraries or code written just for this purpose might be much harder to set-up and require external dependencies. Some of them failed early for me and it was caused by those tools not being optimized for large datasets.