AWS Data Pipeline DynamoDB to S3 503 SlowDown Error

0
votes

We have a Data pipeline that does a nightly copy of our DynamoDB to S3 buckets so we can run reports on the data with Athena. Occasionally the pipeline will fail with a 503 SlowDown error. The retries will usually "succeed" but create tons of duplicate records in S3. The DynamoDB has On-Demand read capacity and the pipeline has 0.5 myDDBReadThroughputRatio. A couple of questions here:

  1. I assume reducing the myDDBReadThroughputRatio would probably lessen the problem, if true does anyone have a good ratio that will still be performant but not cause these errors?

  2. Is there a way to prevent the duplicate records in S3? I can't figure out why these are being generated? (possibly the records from the failed run are not removed?)

Of course any other thoughts/solutions for the problem would be greatly appreciated.

Thanks!

1
amazon-web-servicesamazon-s3amazon-dynamodbamazon-data-pipeline
From what I can tell issue is that the on-demand capacity scales up to be twice the peak traffic on the table. Since we are using .5 throughput literally any other traffic on the table will increase the peak, which will increase the throughput. Eventually, given enough other traffic, the throughput becomes too high and causes the 503 by doing more than 3,500 writes per path per second. I will look into lowering the throughput for the short term solution, and look into the links dDarkLORD provided for a better long term solution. – WillK

1 Answers

1
votes

Using AWS Data Pipeline for continuous backups is not recommended.

AWS recently launched a new functionality that allows you to export DynamoDB table data to S3 and can be further analysed by Athena. Check it out here

You can also use Amazon glue to do the same (link).

If you still want to continue to use data pipeline, then the issue seems to be happening due to S3 limits being reached. You might need to see if there are other requests also writing to S3 at same time OR if you can limit the request rate from pipeline using some configuration.