Mongodb - 16Mb limit with aggregation during unwind

Question

I've got a collection on which I want to perform an aggregation. Weeks ago, everything worked perfectly, and since I've added some documents (with the same schema). However, when I try to aggregate my collection, I have the 16Mb limit of a BSON document exception that is thrown.

The most surprising is that I tried to identify when the limit is reached in the aggregation, and it is reached at the first step, where I simply unwind part of the document.

My question is then, how a stored document (which is by nature of size less than 16Mb) can become more than 16Mb after unwinding ? Note that the exception is thrown when I try to insert the resulting cursor into another collection with the InsertBatch method. I tried also to enumerate in the cursor but I have the same exception, the same if I try to access the first document.

[Edit]: I'm using version 2.6.1 of MongoDB and version 1.9.2 of the C# driver

[Edit]: The documents I use are similar to

{
    "_id" : ObjectId("53ff90e8dcb48a09f090f291"),
    "Val1" : 10
    "Val2" : true,
    "Mesure" : {
        "_id" : ObjectId("53ff90e8dcb48a09f090f284"),
        "Calibration" : {
            "CalibrationDate" : ISODate("2014-08-28T16:29:59.335-04:00"),
            "Expected" : [ 
                {
                    "Bin" : 12,
                    "PositionCalibrated" : 0,
                    "PositionExpected" : 0,
                    "PositionMax" : 0,
                    "PositionMin" : 0,
                    "PositionStd" : 0,
                    "PositionTab" : [ 
                        0, 
                        0, 
                        0
                    ],
                    "PositionTolerance" : 1
                }, 
                {
                    "Bin" : 11,
                    "PositionCalibrated" : 406.113044449032,
                    "PositionExpected" : 401.1,
                    "PositionMax" : 406.113044449032,
                    "PositionMin" : 406.113044449032,
                    "PositionStd" : 5.684341886080802e-014,
                    "PositionTab" : [ 
                        406.113044449032, 
                        406.113044449032, 
                        406.113044449032
                    ],
                    "PositionTolerance" : 1
                }, 
                {
                    "Bin" : 7,
                    "PositionCalibrated" : 454.9231746931026,
                    "PositionExpected" : 431.1,
                    "PositionMax" : 454.9231746931027,
                    "PositionMin" : 454.9231746931027,
                    "PositionStd" : 5.684341886080802e-014,
                    "PositionTab" : [ 
                        454.9231746931027, 
                        454.9231746931027, 
                        454.9231746931027
                    ],
                    "PositionTolerance" : 1
                }
            ],
            "_id" : ObjectId("53ff9143dcb48a09f090f2ae")
        }
    },
    "PositionMeasured" : 407.23645821
}

I use the aggregation pipeline with the C# driver. I want to do several aggregation steps (unwind, sort, group, project etc...) but the first step is to unwind those elements. The code used is as follow :

var UnwindCalib = new BsonDocument("$unwind", "$Mesure.Calibration.Expected");
var pipeline = new[] { UnwindCalib };
var args = new AggregateArgs();
args.Pipeline = pipeline;
args.AllowDiskUse = true;
var result = myCollection.Aggregate(args);
resultsCollection.InsertBatch(result); <----- I've got an error here.

The goal after unwinding is to match an expected item in the calibration to the current item.

To calculate the size of output after unwinding, you can approximate it to be size of single doc * total number of elements in the unwinded array. — displayName
What aggregation query are you using? Isn't it simply unwinding multiple documents? — Kenneth
Please include the pipeline and a description of how you know which pipeline stage is causing the problem. — wdberkeley

AxxE AxxE · Accepted Answer · 2014-08-29T14:29:40

I believe the Mongo DB agggregate operation will return a cursor only if you pass a cursor:{} argument to it. The cursor aggregation option is documented here. Working with cursors returned by aggregate operations using the C# driver is discussed in this thread

Also, if your intention at the first stage is to store the aggregation result to another collection you should use {$out:[collection_name]} as the last stage in your aggregation pipeline. The mongodb documentation for the $out operator can be found here. This operator circumvents the 16MB document size limit since it's an in-DB operation and is handled accordingly by Mongo DB.

Mongodb - 16Mb limit with aggregation during unwind

1 Answers