Is there a way to sync table data from DynamoDB to Redshift?

1
votes

We are currently using Amazon RDS for our database and we are planning to move to DynamoDB to scale our messaging DB. But a lot of analytics has been setup on MySQL tables which we want to keep using.

What is the best way to sync data from DynamoDB to Redshift ?

1
amazon-dynamodbamazon-redshiftdatabase-replicationtungsten-replicator
you cannot sync, but you can use redshift COPY command. – Jon Scott
@JonScott Can we periodically copy data to redshift COPY command, including create,update and delete changes ? – chaithu
I am confused. You say you wish to move from Redshift to DynamoDB, then you mention MySQL tables, then you ask how to sync from DynamoDB to Redshift. Could you please clarify what transfer you are actually asking about? – John Rotenstein
@JohnRotenstein read again - OP says "We are currently using Amazon RDS for our database and we are planning to move to DynamoDB" I assume RDS mysql is current. – Jon Scott
Oops! My bad, thank you. So I think that means the OP is wanting to move from RDS to DynamoDB and they have an existing method of transferring data from RDS to Redshift, and they want to change that to transferring from DynamoDB to Redshift. Is that a correct reading? – John Rotenstein

1 Answers

0
votes

For this kind of problems (synchronization of source database into Redshift) you have basically two common solutions:

  • (A) create application that periodically polls input database, detects delta and ingest detected delta data into Redshift
  • (B) use AWS Kinesis Firehose

Option B is much easier to implement, as you basically use dedicated AWS service for this purpose. However it is suitable only for source databases supported by Firehose. Luckily for you, DynamoDB is supported as input source for Firehose. Also you need to calculate an extra cost associated with Kinesis itself, Lambda used for record conversions, etc. to be sure that this option is valid for you from economical point of view. See this article to see how you can connect DynamoDB streams with Kinesis using Firehose.

Option A is usually a valid solution for source databases not supported by Firehose.