Storing timeseries data in DynamoDB

Question

I am trying to store a bunch of network measurements in DynamoDB and I am not sure of the correct partition key and range key. My data will look like this:

{
count: 33,
total: 45,
timestamp: 21231133,
data: {
    key1: value1,
    key2: value2,
    key3: value3,
    key4: value4,
    key5: value5}
}

I need to be able to retrieve data that has a timestamp between timestamp1 and timestamp2. I will be storing the data every second, and retrieving at a slower interval (maybe every 30 seconds).

I need to avoid hot-partitions and still be able to quickly query the data. I would not want to do a table scan.

Thanks,

Keep source of data as Primary-Key, Timestamp (in nanoseconds if you're getting lots of data per second) as SortKey, this would allow you to query data based on every device independently. If the use-case was also to get all the data between a time-frame, you could use a GSI on an attribute that defines the type of device that way you could query [for type-x give me data between StartTime to Endtime]. So every packet would have DeviceID (primary-key), Timestamp (SortKey), DataType(GSIPK), and use Timestamp as the GSISK — user2967920

notionquest notionquest · Accepted Answer · 2016-06-29T09:02:30

Based on the query pattern mentioned in the question, I think "timestamp" attribute should be a PARTITION key. You can use BETWEEN operator when you define the attribute as String , Number or Binary.

On this table data, I don't see any attribute that can be used as RANGE key unless you have any other query pattern or use case.

BETWEEN : Greater than or equal to the first value, and less than or equal to the second value. AttributeValueList must contain two AttributeValue elements of the same type, either String, Number, or Binary (not a set type). A target attribute matches if the target value is greater than, or equal to, the first element and less than, or equal to, the second element. If an item contains an AttributeValue element of a different type than the one provided in the request, the value does not match. For example, {"S":"6"} does not compare to {"N":"6"}. Also, {"N":"6"} does not compare to {"NS":["6", "2", "1"]}

Storing timeseries data in DynamoDB

1 Answers