copy payload data from dynamodb to redshift

1
votes

Newbee to AWS.

I want to use COPY command to import table from dynamoDB to Redshift. But I occurred the error message such as "Invalid operation: Unsupported Data Type: Current Version only supports Strings and Numbers". Or I can only have values in some columns and others(more important one, such as sensor value in paylaod) are null.

In dynamoDB, hashkey and rangekey are String, but the payload is JSON format, how I can COPY this payload to Redshift? the documentation in AWS didn't provide a detail solution.

1
jsonamazon-web-servicesamazon-dynamodbamazon-redshift
github.com/awslabs/lambda-streams-to-firehose – systemjack

1 Answers

2
votes

COPY command can be used to copy data from DynamoDB table which has scalar data types (i.e. STRING and NUMBER).

If you have any attributes in DynamoDB table which has different data types (i.e. Map, List, Set etc.), the COPY command would fail (i.e. it is not supported at the moment).

Only Amazon DynamoDB attributes with scalar STRING and NUMBER data types are supported. The Amazon DynamoDB BINARY and SET data types are not supported. If a COPY command tries to load an attribute with an unsupported data type, the command will fail. If the attribute does not match an Amazon Redshift table column, COPY does not attempt to load it, and it does not raise an error.