MongoDB + Nodejs data fetch failing after multiple requests

1
votes

I am hitting mongodb end point with around 30,000 requests per minute. Everything seems to work fine till 20,000 requests and after that all requests are failing with

t [object Object]. (/home/ubuntu/node_modules/mongodb/lib/mongodb/connection/server.js:530:7) at [object Object].emit (events.js:88:20) at [object Object]. (/home/ubuntu/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:104:15) at [object Object].emit (events.js:70:17) at Socket. (/home/ubuntu/node_modules/mongodb/lib/mongodb/connection/connection.js:411:10) at Socket.emit (events.js:67:17) at Array.0 (net.js:319:25) at EventEmitter._tickCallback (node.js:192:41)

closing the db connection per iteration, FYI, i am connecting to multiple DBs everytime.

Repeating the same process again is having the same consequences.

1
mongodb

1 Answers

1
votes

This is usually a ulimit issue on the server because you are opening too many connections, though you will need to look at the mongod/mongos logs to be sure. If this is the case you will see errors about not being able to allocate resources and/or create new threads. For how to remedy this take a look here:

http://docs.mongodb.org/manual/administration/ulimit/

Also, closing the connection, depending on the driver you are using can be a very bad idea and may be the root cause of your problems (especially if you have your ulimits set appropriately). Most of the drivers these days, including node I believe have a connection pool that manages connections appropriately and allows for re-use. By closing them explicitly you may actually be causing old connections to build up in the short term.