Slow Mongoose Requests

2
votes

I've been having really slow Mongoose queries. I've narrowed it down to either being the way Mongoose creates objects or the actual latency in the db server requests. I have barely any data in the database (< 1000 objects in each of the 4 collections) and ran queries (which were similar to the queries in the product) manually and profiled them and they all ran < 2-4 ms.

Two big questions which I have not been able to figure out by searching online / looking at the docs:

  1. How do I get the raw queries that Mongoose is running on Mongo?
  2. How many trips to the db server does a call make - for example if I have a query with a couple populate calls, a sort call, and a nested populate call in the callback function how many trips does that make to the server.

Here's some of the latency times I'm seeing:

Login
localhost + localDB = 58ms
localhost + mongoHQ = 127ms
heroku + mongoHQ = 287ms

Get Data
localhost + localDB = 281ms
localhost + mongoHQ = 1657ms
heroku + mongoHQ = 2190ms

Update 1

I figured out how to log the queries and checked up on a bunch of them. I think its coming down to one query with a bunch of populate calls on it - I noticed that in generating the population part of the query, Mongoose outputs a Mongo query like:

users.find({ _id: { '$in': [ ObjectId("531ec0e17c0b16a82be4f506"), 
ObjectId("531ec0e17c0b16a82be4f506"), ObjectId("531ec0e17c0b16a82be4f506") ...

in which the ids are repeated multiple times (in this case several hundred times) - I think this is the cause for the slowdown and testing the code I found that each populate call adds about .3s-.5s to the request time. The populate calls are also running on an array field - this also keeps the number of ids to be queried for in the 100s - low 1000 (of duplicate).

Is there a way to remove the duplicate entries when doing a populate call?

1
node.jsmongodbmongoose
Can you connect with the mongo console to one of the instances and see if they are sluggish when directly connected? I can't imagine that the times you posted could not be explained by mongoose object population unless it was many tens of thousands. – WiredPrairie
@WiredPrairie it seems fine - also when I ran the login user test (see above) it was pretty fast when connecting to the remote db 127ms – N V
@WiredPrairie I did some investigation, do you think it could be because there are a lot of arguments in the populate call? – N V
It could be ... you might just want to roll your own solution at least experimentally (gather the _ids, without dups and see what happens). – WiredPrairie
Assemble the objects myself? kk I'll try it out – N V

1 Answers

5
votes

Try changing your queries to use the "lean" option and see if the amount of time reduces. That will bypass the mongoose entity creation.

http://mongoosejs.com/docs/api.html#query_Query-lean