MongoDB sharded cluster 25 slower than standalone node

5
votes

I'm confused by the situation and trying to fix this for a couple of days now. I'm running 3 shard on top of three 3-members replica sets (rs0, rs1 and rs2). All is working so far. Data is distributed over the 3 shards as well as cloned within the replica sets.

BUT: importing data into one of the replica set works fine with constantly 40k docs/s but by enabling sharding slows the entire process down to just 1.5k docs/s.

I've populated the data via different methods:

  • generated some random data in the mongo shell (running in my mongos)
  • JSON data import via mongoimport
  • MongoDB dump restore from another server via mongorestore

All of them result in just 1.5k doc/s which is disappointing. The mongod's are physical Xeon boxes with 32GB each, the 3 config servers are virtual servers (40 GB HDD, 2 GB RAM, if that matters), the mongos is running on my app server. By the way, the value of 1.5k inserts/s doesn't depend on the shard key, same behaviour for a dedicated shard key (single field key as well as compound key) as well as hashed shard key on _id field.

I tried a lot, even reinstalled the entire cluster twice. The question is: what is the bottleneck in this setup:

  • config servers running on virtual server? -> shouldn't be problematic due to the low resource consumption of config servers
  • mongos? -> running multiple Mongos on a dedicated box behind HAproxy might be an alternative, haven't tested that yet
2
performancemongodbreplicationsharding
Try to stop the balancer for the duration of this massive bulk update and see if that speeds things up. Also if can and the data allows it, try using a targeted shard technique such as Tag Aware Sharding – Ori Dar
When you're loading data into a single server, you're probably using large batches that amortize the cost of locking, journalling, network overheads, etc. When you load through a mongos, the mongos needs to break up your batches into smaller batches that go to each shard. The way it does this is inefficient (github.com/Tokutek/mongo/issues/912 explains more). A way around this, if this is your issue, is to make sure each batch of documents from your application all have the same shard key, somehow. – leif
OK, what's confusing is: collections that are not registered by sh.shardCollection() to be sharded are stored by default at the primary shard. So no routing, load balancing etc. is required for them. So I've created a dummy collection, didn't delate it to be shared and populated it in mongos mongo shell. Result: same slow data inserts like in shared collections. Still 1/25 of the speed when importing same data to one of the replica sets. – ctp

2 Answers

4
votes

Let's do the math first: how big are your documents? Keep in mind that they have to be transferred over the net multiple times depending on your write concern.

May be you are experiencing this because of the indices which have to be build.

Please try this:

  1. Disable all indices except the one for _id (which is not possible anyway, iirc)
  2. Load your data
  3. Reenable indices.
  4. Enable sharding and balancing if not done already

This is the suggested way for importing data into a shared cluster anyway and should speed up your import considerably. Some (cautious !) fiddling with storage.syncPeriodSecs and storage.journal.commitIntervalMs might help, too.

The delay can occur even when storing the data on the primary shard. Depending on the size of your indices, they may slow down bulk operations considerably. You might also want to have a look at the replication.secondaryIndexPrefetch config option.

Another thing might be that your oplog simply gets filled faster than the replication can take place. Problem here: once it is created, you can not increase it's size. I am not sure wether it is safe to delete and recreate it in standalone mode and then reshare the replica set, but I doubt it. So the safe option would be to have the instance actually leave the replica set, reinstall it with a more appropriate oplog size and add the instance to the replica set as if it were the first time. If you don't care for the data, simply shut the replica set down, adjust the oplog size in the config file, delete the data dir and restart and reinitialize the replica set. Thinking of your problem twice, this sounds like the best bet to me, since the opllog isn't involved in standalone mode, iirc.

If you still have the same performance issues, my bet is on problems with disk or network IO.

You have a fairly standard setup, your mongos instance is running on a different machine than your mongod (be it a standalone or the primary of a replica set). You might want to check a few things:

  1. Name resolution latency for resolving the name of your primary and secondary shards from the machine running your mongos instance. I can not count the times installing nscd improved performance for various operations.
  2. network latency from your mongos instance to your primary shard. Assuming you have a firewall between your AppServer and your cluster, you might want to talk to the respective administrator.
  3. In case you are using external authentication, try to measure how long it takes.
  4. When using some sort of tunneling (e.g. stunnel or encryption like SSL/TLS), make sure you disable name resolution. Please keep in mind that encrypting and decrypting may take a relatively long time.
  5. Measure random disk IO on the mongod instances
1
votes

I was facing a similar performance issue. What helped to solve the performance issue was I ended up setting the mongod instance that was running on the same host as the mongos as the primary shard.

using the following command:

mongos> use admin
mongos> db.runCommand( { movePrimary: "mydb", to: "shard0003" } )

After making this change (without touching the load balancer or tweaking anything else), I was able to load a relatively large dataset (25 million rows) using a loader I had written, and the entire procedure took about 15 minutes instead of hours/days.