mongodb sharding auto-increment shard-key

1
votes

After reading the documentation on sharding, shard-keys and chunk migration i still cant grasp one concept.

Anyone trying to describe the concept of why NOT to choose a auto-increment shard-key tells me that this is due to the result that mongo will always keep writing to the same shard, and therefore this will increase the load since that shard will both handle consecutive writes and also migrate chunks.

My question is, why is this the case? Why will an auto-increment value result in write requests always being routed to one specific shard? Isnt the whole point of sharding that mongos should be aware of what ever shard that is "least" balanced and instead write to this shard, or am i understanding it wrong?

thanks in advance

2
mongodbsharding
In laymans terms because an auto increment value linearly increases so the shard_key being a range means that it will always write to the end of that key, i.e. the last server, so i.e. 4 comes after 3 at all times, there is no other way to read that and so 4 will (unless the server containing 3 is full) write to the same server as 3. - Sammaye
If i understand it correctly the keyspace is somewhat distributed of the amount of shards, hence mongos will always write to the shard coherent to that portion of the keyspace? - dunn less
It depends on the write, if you update yes it will wrtie to the distributed shard it is on but if it is an insert then it will basically plonk everything onto one server and then attempt to rebalance. So this isn't so bad if you don't expect a high frequency of inserts on that collection. - Sammaye

2 Answers

2
votes

The current shard mechanism has this problem. Basically, each shard will serve data whose key lays in a consecutive ranges. If we choose an auto-incremental key, all write-operations will be routed to only one shard, which serves data with key larger than all other shards.

This is a problem for collection with auto-incremental key. Fortunately, we can choose any attribute as sharding key. For most cases, we are not forced to use "_id" as sharding key. If object has a suitable attribute, such as "user name" for a blog application, we can use it as the sharding key. If there is no suitable attribute, we can add one attribute column to each object as the sharding-key, this attribute can be computed using some simple hash algorithm. For example, if we have an auto-incremental _id, we can compute the sharding key as simple as:

sharding_key = _id % 257

Above hash should be good enough before you have more than 257 shards.

BTW, the auto-generated ObjectId is not a good choice for sharding-key, cause it's time-based.

Also, there is a new feature in MongoDB 2.3 to support hash-key (See https://jira.mongodb.org/browse/SERVER-2001 and MongoDB 2.4 release note).

2
votes

The problem is Mongo can't determine key ranges for sharding if you use monotonically increasing keys. Here's a sample:

Suppose you have a collection with keys 10,20,30,40,50,60 If mongo have to create two shards, it may assume the keys range as: [10,30] and [31,60] (or similar). But if you continue to write bigger keys, they always will go to second range. Mongo will adjust ranges, but it can never know what the next key will be, and it will always get into the last range. On the other hand, if you use some key with good distribution, your write sequence will look more like: 10, 60, 30, 40, 50... And just after writing two first keys, mongo will create the above ranges, and your next keys will fit into either first or second. This will result in sharing performance among mongos and also, will not force mongo to do rebalancing.