MongoDB - shard collection by hashed index on custom _id field

2
votes

Problem: How to shard collection by hashed index on custom _id field?

Problem description:

  • I need to store pairs url => my_value in MongoDB
  • Url must be unique
  • I will execute a lot of queries to check, if i already have document with such url by matching {_id : md5(url_to_check)}
  • Collection will be huge (billions of pairs url => my_value), so i want to shard it by url.

Solution, i consider:

Create collection with such fields:

  • _id : md5(url)
  • url : url
  • value : my_value I don't create any index. _id is default indexed by mongo

Questions:

  • I would like to shard collection by _id. Hashed shared key would be perfect, but do i have to create hashed shard key or can i just shard by regular _id key? I insert to _id already computed md5 by myself.
  • What do you think about storing in _id not-hashed url and query by it? I would use less space (don't have to storedmd5(url)), but sharding will by on bigger text field and index will be on biger string (usualy url has more than 32 sings)
  • What is it the best solution to solve such problem? Best means for me fast queries and use as less space for indexes, as it is required?
1
mongodbindexingsharding

1 Answers

4
votes

I would like to shard collection by _id. Hashed shared key would be perfect, but do i have to create hashed shard key or can i just shard by regular _id key? I insert to _id already computed md5 by myself.

A hashed shard key is intended to be used with fields that increase monotonically (like ObjectId() values or timestamps) in order to provide more uniform distribution of write load across your shards. If you've already hashed your _id values (or a field you want to shard on) you can use this as your shard key instead of requesting the server to calculate this for you.

FYI, MongoDB (as at 2.6) uses md5 to compute a hashed shard key, so effectively you are doing the same work in your application code already and making more effective use of the _id index. With your use case of a pre-hashed _id value you only need a single _id index as compared to two indexes (the default index of {_id:1} plus an extra hashed index {_id:hashed}).

What do you think about storing in _id not-hashed url and query by it? I would use less space (don't have to storedmd5(url)), but sharding will by on bigger text field and index will be on biger string (usualy url has more than 32 sings)

If index size is a concern, the smaller precomputed values will definitely save you space in the _id index (especially if you are storing billions of urls and only want to find documents by the md5 hash).

What is it the best solution to solve such problem? Best means for me fast queries and use as less space for indexes, as it is required?

Best solution is highly subjective, but it seems like this is a reasonable solution given what you've shared of your use case.

It's worth noting that any hashed namespace can potentially have collisions, so you may want to consider the collision resistance of your hash algorithm relative to the namespace. Although collisions should be extremely unlikely, with the hash value as your _id you will only store the first url observed for any hash collisions (or have to add something less efficient, like a comparison of the document url vs original url you were expecting).