dynamodb vs aws s3 for timeseries data

Question

My data model is the following: Timestamp, value

This is currently being stored in a csv file in s3, where the client downloads and uses it to append value onto some data, with the model: Timestamp, name

the final file is the Timestamp, name, value

Would it be faster to store all of the data model in a dynamodb table and lookup each one via timestamp?

My problem was that we would be looking up for 100-20k records and I am not sure how dynamo would handle this. If the whole file is on the client side, which is about 3MB big, then it can be done locally.

This file is 3MB now, however it will grow over time, via a scheduled lambda function. I do not care about strong writes that much, just need it to be eventually consistent.

Also, if data was in dynamo, then I could append value in lambda instead of doing it on client side. If dynamo is too slow, this could timeout lambda however.

Why not use an actual timeseries database? blog.timescale.com/… — OneCricketeer
That is the correct solution and the one I eventually went with. The reason I discarded it before was due to the fact that incorrect modelling could burn through my RCUs and WCUs in no time and add latency. This is the correct solution however, if latency occurs I will use Dax. Thanks for the blog link cricket — WeCanBeFriends
@WeCanBeFriends i am interested in what solution you end up with, now dynamodb supports timeseries, but at the time we have had unexpected costs using it in my company. We ended up gathering data in csv files in s3, one per day, and loaded on the fly by a Lambda call when needed... — Cinn

sid8491 sid8491 · Accepted Answer · 2018-01-31T06:51:32

I have used DynamoDb for the similar use case, and it was working perfectly fine, it took far less than a second to put the item in DynamoDb table.
And it would be faster if you put both the Lambda function and the DynamoDb in the same region.

Also, think about the structure of DynamoDb table (partition key and sort key) according to your use case for optimized transactions.

dynamodb vs aws s3 for timeseries data

1 Answers