Export | Import DynamoDB only with AWS CLI

3
votes

My goal is to have simple tool for export dynamodb to local file(json/csv) only with aws cli or less 3th party as it's possible. The size of my tables are around 500mb.

I'm doing the research for few days now and I saw very nice projects like dynamodump and another SDK implementations. Also I test the Datapipeline from Amazon, but for task that I'm targeting want simple aws cli.

Let's say that I already have map for the table such as indexs and attributes.Now I want to import just the Items that I have. For me the best solution for the implementation that I have will be something like this:

  1. Exporting the table from aws to local file: aws dynamodb scan --table-name SOMETABLENAME --output json | jq '.Items[]' > SOMETABLENAME.json

  2. Rendering the file with things like sed, jq, awk ...etc. To get proper json format whom can be imported to aws-dynamodb

  3. Import the Items from SOMETABLENAME to aws-dynamodb with:

    aws dynamodb put-item --table-name SOMETABLENAME --item file://SOMETABLENAME.json

2
amazon-dynamodbaws-cli
I'm not sure what the question is. It looks like you already figured out the command line to export a table using the AWS CLI (using aws dynamodb scan) as well as for imports, using aws dynamodb put-item. Are you just looking for confirmation? - Mike Dinescu
only AWS-Cli is not going to help you here because scan will return the json with limited records so is you want to export entire table then you need create script which iterates and store data in local files - Chandani Patel

2 Answers

0
votes

DynamoDB released a new feature to export your data to an S3 bucket. It's a native, server-less solution that can be used via the CLI, as you desire - see:

https://docs.aws.amazon.com/cli/latest/reference/dynamodb/export-table-to-point-in-time.html

That will help you achieve #1 without the intermediate scan, but unfortunately, as of Nov 2020, there's no native import functionality to get it to a new table.

If all you're interested in is getting the data from one table to another, you can use PiTR:

https://docs.aws.amazon.com/cli/latest/reference/dynamodb/restore-table-to-point-in-time.html

-1
votes

You can export the data using aws dynamodb scan.

aws dynamodb scan --table-name table_name --region aws_region_name --max-items max_items --output json > ./export.json

However,DynamoDB has a 1MB limit on the amount of data it will retrieve in a single request. If you hit the 1MB limit with a Scan, it will return a "NextToken" key in the response. You need to scan it again with NextToken, until all the data have been exported.

aws dynamodb scan --table-name table_name --region aws_region_name --max-items max_items --starting-token nextToken --output json > ./export2.json

Regarding the import, BatchWriteItem can write up to 16 MB or 25 put or delete requests. You can set the max_items to 25 or split it using jq.

You can convert the json to BatchWriteItem cli via jq.

cat filepath | jq "{\"table_name\": [.[] | {PutRequest: {Item: .}}]}" > BatchWriteItem.txt

Finally, you can insert the to dynamodb using cmd.

aws dynamodb batch-write-item --request-items file://filepath

It's better to do with shell script if the data is huge. More details: https://github.com/mickey2012ex/dynamodb-data-migrator

The best solution is AWS Data Pipeline, if the region is supported