AWS DynamoDB to Kinesis without DynamoDB Stream

1
votes

We have a requirement to stream data from DynamoDB tables to Kinesis stream for event monitoring. We started looking into DynamoDB stream but the issue is DynamoDB stream is not encrypted and we can't have any unencrypted data in our solution anywhere. What is the other approach in serverless to stream data from DynamoDB to Kinesis? I don't want to stand up a server to use DynamoDB adapter. Thanks

2
amazon-dynamodbamazon-kinesis
If the data must be encrypted everywhere, then you must be doing some client side encryption before you send it to dynamodb, right? In that case, the data will still be encrypted in the stream. – Matthew Pope

2 Answers

1
votes

As of now(Sept. 2019), at rest encryption is supported in DynamoDB stream.

DynamoDB encryption at rest provides an additional layer of data protection by securing your data in the encrypted table, including its primary key, local and global secondary indexes, streams, global tables, backups, and DynamoDB Accelerator (DAX) clusters whenever the data is stored in durable media.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EncryptionAtRest.html

0
votes

If you wanted to wanted to use DynamoDB streams this is how you could do it:

Enable DynamoDB steams, but set it to "Keys only". This mode will give only the key attributes of the modified item. Then setup a Lambda to trigger off this DynamoDB stream, this will send batches of keys to your Lambda. You then code the lambda to lookup the key in your DynamoDB database and then push it into Kinesis.

It's not a perfect solution because the data may have been updated again before the Lambda get operation, but it's pretty good depending on the situation.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

If you didn't want to use DynamoDB streams would you either have to have your client application also push the data to Kinesis, or if you can't configure the client application instead don't let anyone talk directly to DynamoDB and instead have them call a lambda synchronous where that Lambda will do the DynamoDB and Kinesis call for you.