Spring Data MongoDB Aggregation $out pipeline

Question

I'm working with Spring data mongoDB and using aggregation to fetch docuemnts.

List<AggregationOperation> operationsList = new ArrayList<AggregationOperation>();

    operationsList.add(Aggregation.unwind("calendarEvent"));
    operationsList.add(Aggregation.match(criteria));
    operationsList.add(getMacroEventProjectionFields());

    if (start <= 0) {
        start = 1;
    }

    operationsList.add(Aggregation.skip(start - 1));

    if (limit > 0) {
        operationsList.add(Aggregation.limit(limit));
    }

    Aggregation aggregation = Aggregation.newAggregation(operationsList);

    AggregationResults<MacroEvent> groupResults = mongoTemplate.aggregate(
        aggregation, mongoTemplate.getCollectionName(KALiEvent.class),
        MacroEvent.class);

The collection contains thousands of records and it was working fine till yesterday. But today it has started throwing the following exception:

Error [exception: aggregation result exceeds maximum document size (16MB)]

I Googled this issue and found that aggregation can only return the results equal to the size of MongoDB document size, which is 16MB. The only workaround I found for this is to use the $out pipeline which does the following:

Takes the documents returned by the aggregation pipeline and writes them to a specified collection. The $out operator must be the last stage in the pipeline. The $out operator lets the aggregation framework return result sets of any size.

But, I just can't figure out the way that how this can be done using Spring data MongoDB and how I can use this $out in my code pasted above.

Are you actually running MongoDB 2.6 or greater on the server? You should be even if you are not. $out is not a general solution ( but also only applies to 2.6 or greater ) as it's intention is to write the output to a collection, rather than a singular BSON document over the wire. The "real" 2.6 and greater solution is to return an "aggregation cursor" instead. This also removes the BSON restriction as the results are in a "Cursor" rather than a single document response. — Blakes Seven

kuk kuk · Accepted Answer · 2016-11-07T11:38:14

In order to avoid that 16 mb exception, I've created my own OutOperation class that performs mongo $out pipeline operation:

package com.myop;

import com.mongodb.BasicDBObject;
import com.mongodb.DBObject;
import org.springframework.data.mongodb.core.aggregation.AggregationOperation;
import org.springframework.data.mongodb.core.aggregation.AggregationOperationContext;

public class OutOperation implements AggregationOperation {

/** 
  the name of the collection where aggregation data will be stored
*/
private String collectionName;

public OutOperation(String collectionName) {
    this.collectionName = collectionName;
}

@Override
public DBObject toDBObject(AggregationOperationContext context) {
    return new BasicDBObject("$out", collectionName);
}
}

Spring Data MongoDB Aggregation $out pipeline

1 Answers