AWSDatapipeline Backup/Restore and Validation

Question

I am new in AWS datapipeline and i need to do backup of dynamoDb to S3 bucket and then restore from that backup back to some restored dyanmoDb table and then validate the records,means check number of records in S3 backup and restored dynamoDb table.

can somebody please let me know how to do this ? I know that there is already template in datapipeline to copyrecords from Dynamo to S3 and S3 to dynamo.

But I wanted from some one experienced,the way to do all Backup,Restore and Validate in same datapipeline, mainly restore and validate steps.

Any help would be valuable

Answer 1

You can basically just combine the two templates together with one after the other to get your desired pipeline as far as backup and restore goes (if you're not sure how to do that, you can set another activity to execute in the data pipeline after the current one finishes).

As for validation, you could in theory do that as part of a shell command activity, but I'd recommend against it. Shell command activities are very complicated to setup and debug, so you're much better off running some other process to take care of validation.

(I'm serious about shell command activities being difficult to work with. You get very little insight into how they are running, what happens during their run, and the status of the run.)