How do you transform DynamoDB Map/List types to through AWS Pipeline to Redshift?

Question

I have two DynamoDB tables, one with a Map data type (JSON) and one with a List data type (list of JSON).

Our current pipeline to Redshift claims these are unsupported data types.

How can I transform these columns to Redshift as varchar(MAX)?

AravindR AravindR · Accepted Answer · 2016-04-19T17:28:21

Can you provide your AWS DataPipeline pipeline id?

Also, take a look at this sample which uses Data Pipeline's HiveActivity and RedshiftCopyActivity to copy data from a DynamoDB table to a Redshift table while performing data conversion using Hive (for data transformation) and S3 (for staging). This sample was motivated by a use case where one wishes to convert the data type of one column to another data type. This sample demonstrates converting a column from binary to base64 string.

https://github.com/awslabs/data-pipeline-samples/tree/master/samples/DynamoDBToRedshiftConvertDataUsingHive

How do you transform DynamoDB Map/List types to through AWS Pipeline to Redshift?

1 Answers