MongoDB sharding problems

Question

Our mongodb server deployed with 2 shards, each has 1 master server and 2 slave servers. The four slave servers run mongo config as proxy, and two of the slave servers run arbiters. But the mongodb coundn't be used now.

I can connect to 192.168.0.1:8000(mongos) and exec queries like 'use database' or 'show dbs', but i cann't exec queries in a choosed database such as 'db.foo.count()', 'db.foo.findOne()'

Here is the error log:

mongos> db.dev.count()
Fri Aug 16 12:55:36 uncaught exception: count failed: {
    "assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520db0a51fa00999772612b9'), authoritative: true }",
    "assertionCode" : 10276,
    "errmsg" : "db assertion failure",
    "ok" : 0
}




Fri Aug 16 11:23:29 [conn8431] DBClientCursor::init call() failed
Fri Aug 16 11:23:29 [conn8430] Socket recv() errno:104 Connection reset by peer 10.81.4.72:7100
Fri Aug 16 11:23:29 [conn8430] SocketException: remote: 10.81.4.72:7100 error: 9001 socket exception [1] server [10.81.4.72:7100] 
Fri Aug 16 11:23:29 [conn8430] DBClientCursor::init call() failed
Fri Aug 16 11:23:29 [conn8430] DBException in process: could not initialize cursor across all shards because : DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: "", init: true, configdb: "10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300", serverID: ObjectId('520d99c972581e6a124d0561'), authoritative: true } @ s01/10.36.31.36:7100,10.42.50.24:7100,10.81.4.72:7100

i can only start on mongos, queries wouldn't be exec if more than 1 mongos run at the same time, error log:

mongos> db.dev.count() Fri Aug 16 15:12:29 uncaught exception: count failed: { "assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520dd04967557902f73a9fba'), authoritative: true }", "assertionCode" : 10276, "errmsg" : "db assertion failure", "ok" : 0 }

did it work before? did you change anything in the configuration? How did you set-up your replica-sets? — Tilo
From the server that is running mongos, can you try connecting to each of the mongods (configs and masters) directly via the mongo shell as a first test? Also, why are you using a Master/Slave configuration over Replica Sets? — Dylan Tong
i can only start on mongos, queries wouldn't be exec if more than 1 mongos run at the same time, error log: mongos> db.dev.count() Fri Aug 16 15:12:29 uncaught exception: count failed: { "assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520dd04967557902f73a9fba'), authoritative: true }", "assertionCode" : 10276, "errmsg" : "db assertion failure", "ok" : 0 } — Sun Pengfei
It is basically saying one of your shards is acting up and one of your comments: "it works well before and be like this since a server crashed and reboot" makes me suspecious, have you checked for corruption on that shard? Have you tried replaying the journal? Have you checked the servers own hardware is ok? — Sammaye

Tilo Tilo · Accepted Answer · 2013-08-16T05:25:09

Could you please clarify if your set-up was working before, if you are just setting it up now?

To repair your MongoDB, you might want to follow this link: http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/

References

MongoDB Documentation : Deploying a Shard-Cluster

MongoDB Documentation : Add Shards to an existing cluster

Older, outdated(!) info:

YouTube Video on Setting-up Sharding for MongoDB

Corresponding Blog on blog.serverdensity.com

MongoDB sharding problems

1 Answers