I want to know how we can load data from DynamoDB into Redshift.
According to the documentation DynamoDB is NoSQL and Redshift is RDBMS.
So how I can handle unstructured data in a normalised way?
When do I need to normalise the data?
I want to know if Redshift keeps full data or transformed data.
I want to know the best way to load incremental Data.
Can anyone suggest the steps for this process?
Loading Data from DynamoDB
The Amazon Redshift COPY command can be used to load a DynamoDB table into a Redshift table. This will load the complete DynamoDB table into Redshift.
See documentation: Loading Data from an Amazon DynamoDB Table
Column names are mapped, and only columns that have matching column names are loaded.
Loading incremental data
To perform an incremental load (eg only where Country='USA'), first load the complete table into a temporary table, then perform normal INSERT SQL commands in Redshift to insert/copy the desired data.
See:
Normalization, foreign & primary keys
DynamoDB is a NoSQL database, so there are no relational concepts between tables and no foreign keys.
When creating tables in Redshift that will receive your data from DynamoDB, you can specify Foreign Keys. These are not enforced by Redshift, but they are used by the query optimizer.
Once data has been imported into Redshift, you can perform relational queries (eg using JOIN) between tables.
Your data does not need to be normalized. In fact, Data Warehouses such as Redshift are often loaded with wide tables and duplicated data that make it easier to query data with fewer JOINS.
