MongoDB Sharding Policy

Question

I am trying to understand the following behavior displayed by my sharding setup. The data seems to only increase on a single shard as I continuously add data. How does MongoDB shard or distribute data across different servers? Am I doing this correctly? MongoDB version 2.4.1 used here on OS X 10.5.

enter image description here

As requested, sh.status() as follows:

mongos> sh.status()
sharding version: {
    "_id" : 1,
    "version" : 3,
    "minCompatibleVersion" : 3,
    "currentVersion" : 4,
    "clusterId" : ObjectId("52787cc2c10fcbb58607b07f") }
shards:
    {  "_id" : "shard0000",  "host" : "xx.xx.xx.xxx:xxxxx" }
    {  "_id" : "shard0001",  "host" : "xx.xx.xx.xxx:xxxxx" }
    {  "_id" : "shard0002",  "host" : "xx.xx.xx.xxx:xxxxx" }
 databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "newdb",  "partitioned" : true,  "primary" : "shard0001" }
            newdb.prov
                    shard key: { "_id" : 1, "jobID" : 1, "user" : 1 }
                    chunks:
                            shard0000       43
                            shard0001       50
                            shard0002       43

Asya Kamsky Asya Kamsky · Accepted Answer · 2013-12-03T03:25:32

Looks like you have chosen a very poor shard key. You partitioned along the values of { "_id" : 1, "jobID" : 1, "user" : 1 } - this will not be a good distribution for inserts since _id values are monotonically increasing since you are using ObjectId() values for _id.

You want to select a shard key that represents how you access the data - it doesn't make sense that you have two more fields after _id - since _id is unique the other two fields are never going to be used to partition the data.

Did you perhaps intend to shard on jobID, user? It's hard to know what the best shard key would be in your case, but it's clear that all the inserts are going into the highest chunk (top value through maxKey) since every new _id is a higher value than the previous one.

Eventually they should be balanced to other shards, but only if the balancer is running, if all your config servers are up and if secondaries are caught up. Best to pick a better shard key and have inserts be distributed evenly across the cluster from the start.

MongoDB Sharding Policy

1 Answers