DynamoDB table structure

Question

We are looking to use AWS DynamoDB for storing application logs. Logs from multiple components in our system would be stored here. We are expecting a lot of writes and only minimal number of reads.

The client that we use for writing into DynamoDB generates a UUID for the partition key, but using this makes it difficult to actually search.

Most prominent search cases are,

• Search based on Component / Date / Date time
• Search based on JobId / File name
• Search based on Log Level

From what I have read so far, using a UUID for the partition key is not suitable for our case. I am currently thinking about using either / for our partition key and ISO 8601 timestamp as our sort key. Does this sound reasonable / widely used setting for such an use case ?

If not kindly suggest alternatives that can be used.

Answer 1

• Using UUID as partition key will efficiently distribute the data amongst internal partitions so you will have ability to utilize all of the provisioned capacity.
• Using sortable (ISO format) timestamp as range/sort key will store the data in order so it will be possible to retrieve it in order.

However for retrieving logs by anything other than timestamp, you may have to create indexes (GSI) which are charged separately.

Hope your logs are precious enough to store in DynamoDB instead of CloudWatch ;)

Answer 2

In general DynamoDB seems like a bad solution for storing logs:

• It is more expensive than CloudWatch
• It has poor querying capabilities, unless you start utilising global secondary indexes which will double or triple your expenses
• Unless you use random UUID for hash key, you are risking creating hot partitions/keys in your db (For example, using component ID as a primary or global secondary key, might result in throttling if some component writes much more often than others)

But assuming you already know these drawbacks and you still want to use DynamoDB, here is what I would recommend:

• Use JobId or Component name as hash key (one as primary, one as GSI)
• Use timestamp as a sort key
• If you need to search by log level often, then you can create another local sort key, or you can combine level and timestamp into single sort key. If you only care about searching for ERROR level logs most of the time, then it might be better to create a sparse GSI for that.
• Create a new table each day(let's call it "hot table"), and only store that day's logs in that table. This table will have high write throughput. Once the day finishes, significantly reduce its write throughput (maybe to 0) and only leave some read capacity. This way you will reduce risk of running into 10 GB limit per hash key that Dynamo DB has.

This approach also has an advantage in terms of log retention. It is very easy and cheap to remove log older than X days this way. By keeping old table capacity very low you will also avoid very high costs. For more complicated ad-hoc analysis, use EMR