Selecting partition key when message does not have right property to diversify documents

0
votes

I have one application that posts messages to Cosmos DB by reading messages from another application database. The only information I can get from other app is documentId i.e. primary key from the app database and the body of the message. Structure is something as below:

{
   "id":<documentid>,
   "body":<body picked up from App>,
   "Timestamp":<today's date time>
}

Body contains simple text message like "Hello World! Today is Wednesday". I have the following requirements: 1. Need to query document by documentId. 2. Need to query documents between two date/time stamps.

  • DocumentId is unique and no duplicate values are allowed.

In this scenario, how can we identify a partition key for the container to retrieve documents easily by the two criterion specified above?

Any inputs are highly appreciated.

2
azure-cosmosdbazure-cosmosdb-sqlapidata-partitioning
Is this a read heavy or write heavy application? - Hasan Savran
Hi Savran, it's both heavy write (~20 million messages/day) and read (will be queried and sent to the interested customer queues) - Seasoned Developer

2 Answers

0
votes

For querying by id: If you know the id, you can do a direct read (vs a query). This requires knowing both the id and the partition key; if this is your primary method of retrieval, you can set your partition key to /id and this becomes the most straightforward approach (and the least expensive, RU-wise).

For querying between timestamps: You'll need to rethink partitioning, since you'd end up doing cross-partition queries if using /id as your partition key. There's really no "right" answer to this part, as we just don't know enough about your app and your query patterns (or volume of data in terms of transaction rate, size, etc). Just know that, if you partition your data, you'll ideally want to limit your queries to be within a single logical partition; otherwise you'll need to perform cross-partition queries, which will cost more RU.

0
votes

If I were you, I would take some part of the timestamp out as string and use it as partition key. For example if timestamp is "4/15/2020 1:20 PM" Create a new property with value "4/15/2020' and make that new property the partition key. That will be sure that you have 20 million messages in each partition. You will have no problems by selecting data by id If you need to search by date, you will know what partition key is since it depends on the timestamp. Only problem I can see is, if you need to retrieve data in date range, you need to hit multiple partitions. Check out my post about synthetic partition keys. https://h-savran.blogspot.com/2019/08/synthetic-partition-keys-in-azure.html I hope that helps!