MongoDB allowDiskUse not working..

16
votes

Experts.

I'm new to MongoDB, but know enough to get my self in trouble.. case in point: 

db.test.aggregate(
[
    {$group: {_id: {email: "$email", gender: "$gender"}, cnt: {$sum: 1}}}, 
    {$group: {_id: "$_id.email", cnt: {$sum: 1}}}, 
    {$match: {cnt: 2}}
], 
    {allowDiskUse : true}
)

and no matter what variations I try, I keep getting the same error ("Pass allowDiskUse:true to opt in"):

Error("Printing Stack Trace")@:0 ()@src/mongo/shell/utils.js:37 ([object Array],[object Object])@src/mongo/shell/collection.js:866 @(shell):7

uncaught exception: aggregate failed: { "errmsg" : "exception: Received error in response from mongo1.mscnet.com:27017: { $err: \"Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.\", code: 16945 }", "code" : 16945, "ok" : 0 }

FYI: I'm using Mongo 2.6

3
mongodbmapreduceaggregation-framework
Are you definitely using a 2.6 shell? What does version() return in the mongo shell? Also, are you using any shell extensions such as Mongo-Hacker? If so, I would try starting the shell using mongo --norc to test without extensions. Your syntax looks correct so I'm suspecting you may inadvertently be using a version of the aggregate() helper that does not support passing options to the aggregation command (which were added in the MongoDB 2.6 shell). - Stennie
I'm using "MongoDB shell version: 2.6.5". Although, I was using a third-party app called RoboMongo (Version 0.8.4). I'm now using the CMD shell with --norc .. and that seems to work.. I guess it was my client. But can you explain something for me?.. The aggregation doesn't happen on my PC (does it?). I don't see any network / cpu usage on my pc. So what is the "client", and how/why does it effect the processing of data? Thank you! - Eyal Zinder
Clients like the mongo shell or Robomongo provide aggregate() helpers to make it more convenient to work with the underlying aggregate command that runs on your MongoDB server. In versions of MongoDB prior to 2.6 the aggregate command did not support any options so older versions of the aggregate() helpers will ignore those. If --norc works for the mongo shell I suspect you have an older version of Mongo-Hacker (I committed a fix in August 2014) - Stennie
I see. So the client sends additional code (helpers) to the server that the server uses to build the execution-path and result - not just the actual command. Thank you again! - Eyal Zinder
mongo shell helpers just manipulate parameters before sending the command to the server (for example, allowing you to pass an aggregation pipeline as either an array or a list of pipeline steps). FYI, you can see the helper code in the mongo shell by invoking without the (), eg: db.test.aggregate - Stennie

3 Answers

20
votes

Please use aggregate query in run command,it will allow to use allowDiskUse tag. 

db.runCommand(
   { aggregate: "test",
     pipeline: [
                {$group: {_id: {email: "$email", gender: "$gender"}, cnt: {$sum: 1}}}, 
                {$group: {_id: "$_id.email", cnt: {$sum: 1}}}, 
                {$match: {cnt: 2}}
               ],
     allowDiskUse: true
   }
)
11
votes

Try writing like this,

db.stocks.aggregate([
                     { $project : { cusip: 1, date: 1, price: 1, _id: 0 } },
                     { $sort : { cusip : 1, date: 1 } }
                    ],
                     {
                       allowDiskUse: true
                     }
                    );

And One more thing,
Your error about maximum document size is inherent to Mongo. Mongo can never return a document (or array thereof) larger than 16 megabytes. I can't tell you why because you didn't mention anything about data size, but it probably means that the document you're building as an end result is too large. Try decreasing the $limit parameter, maybe? Start by setting it to 1, run a test, then increase it and look at how big the result gets when you do that

8
votes

If you are working with m10 or later version of MongoDB Atlas, below is the fix for {allowDiskUse: true}; If you are accessing data from application via mongoose then, it should be syntax as function. For example:

MongoDb Query:

db.collection.user.aggregate([], {allowDiskUse: true})

With Mongoose:

User.aggregate([]).allowDiskUse(true);

This will resolve error from application :)