I was running MongoDB on Amazon AWS, until one day it ran out of space. Then, the database is completely inaccessible,and I tried to shutdown the database. Having restarted the server and installed larger disk, the MongoDB can't be started. I tried to repair the database, and the error is shown like below:
Wed Aug 15 10:08:04 [initandlisten] MongoDB starting : pid=1447 port=27017 dbpath=/var/lib/mongodb/ 32-bit
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data ** see http://blog.mongodb.org/post/137788967/32-bit-limitations ** with --dur, the limit is lower
Wed Aug 15 10:08:04 [initandlisten] db version v1.8.3, pdfile version 4.5 Wed Aug 15 10:08:04 [initandlisten] git version: c206d77e94bc3b65c76681df5a6b605f68a2de05 Wed Aug 15 10:08:04 [initandlisten] build sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37 Wed Aug 15 10:08:04 [initandlisten] ** Wed Aug 15 10:08:04 [initandlisten] ** Wed Aug 15 10:08:04 [initandlisten] need to upgrade database md with pdfile version 559903.1345021735, new version: 4.5 Wed Aug 15 10:08:04 [initandlisten] starting upgrade Wed Aug 15 10:08:04 [initandlisten] md repairDatabase md Wed Aug 15 10:08:04 [initandlisten] md.system.namespaces Assertion failure isOk() db/pdfile.h 259 0x817c27b 0x819121e 0x83cf525 0x83d6bec 0x83f5420 0x82a4fb7 0x83f8680 0x83fcece 0x8401358 0x8401e29 0x8402abd 0x828f755 0x83b3d67 0x83b931c 0x83bb3df 0x81ce25f 0x81ae714 0x8380f31 0x83831ba 0x83d4faf mongod(_ZN5mongo12sayDbContextEPKc+0xcb) [0x817c27b] mongod(_ZN5mongo8assertedEPKcS1_j+0x12e) [0x819121e] mongod(_ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE+0x7a5) [0x83cf525] mongod(_ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE+0x13c) [0x83d6bec] mongod(_ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi+0x830) [0x83f5420] mongod(_ZN5mongo11UserQueryOp5_initEv+0x437) [0x82a4fb7] mongod(_ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE+0x110) [0x83f8680] mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x2ee) [0x83fcece] mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x2e8) [0x8401358] mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x59) [0x8401e29] mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x2d) [0x8402abd] mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1575) [0x828f755] mongod() [0x83b3d67] mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x70c) [0x83b931c] mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x7f) [0x83bb3df] mongod(_ZN5mongo14DBClientCursor4initEv+0x15f) [0x81ce25f] mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x2a4) [0x81ae714] mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x451) [0x8380f31] mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba] mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf] Wed Aug 15 10:08:04 [initandlisten] assertion 0 assertion db/pdfile.h:259 ns:md.system.namespaces query:{} Wed Aug 15 10:08:04 [initandlisten] Assertion: 10290:bad system.namespaces object { $err: "assertion db/pdfile.h:259" } 0x819045e 0x8381181 0x83831ba 0x83d4faf 0x852a30c 0x852b217 0x852e733 0x852f59c 0x85366bb 0xb74a4ce7 0x810b551 mongod(_ZN5mongo11msgassertedEiPKc+0x15e) [0x819045e] mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x6a1) [0x8381181] mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba] mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf] mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x6c) [0x852a30c] mongod() [0x852b217] mongod(_ZN5mongo14_initAndListenEiPKc+0x4b3) [0x852e733] mongod(ZN5mongo13initAndListenEiPKc+0x2c) [0x852f59c] mongod(main+0x6bdb) [0x85366bb] /lib/libc.so.6(_libc_start_main+0xe7) [0xb74a4ce7] mongod() [0x810b551] Wed Aug 15 10:08:04 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object { $err: "assertion db/pdfile.h:259" }, terminating Wed Aug 15 10:08:04 dbexit: Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close listening sockets... Wed Aug 15 10:08:04 [initandlisten] shutdown: going to flush diaglog... Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close sockets... Wed Aug 15 10:08:04 [initandlisten] shutdown: waiting for fs preallocator... Wed Aug 15 10:08:04 [initandlisten] shutdown: closing all files... Wed Aug 15 10:08:04 closeAllFiles() finished Wed Aug 15 10:08:04 [initandlisten] shutdown: removing fs lock... Wed Aug 15 10:08:04 dbexit: really exiting now
I have all *.number and *.ns file completely. But it seems the ns file is corrupted. Any way I could repair the database?
Thanks
