MongoDB fulltext search: Overflow sort stage buffered data usage

Question

I am trying to implement mongo text search in my node(express.js) application.

Here are my codes:

Collection.find({$text: {$search: searchString}}
, {score: {$meta: "textScore"}})
.sort({score: {$meta: 'textScore'}})
.exec(function(err, docs {

//Process docs
});

I am getting following error when text search is performed on large dataset:

MongoError: Executor error: Overflow sort stage buffered data usage of 33554558 bytes exceeds internal limit of 33554432 bytes

I am aware that MongoDB can sort maximum of 32MB data and this error can be avoided by adding index for field we will be sorting collection with. But in my case I am sorting collection by textScore and I am not exactly sure if is it possible to set index for this field. If not, is there any workaround for this?

NOTE: I am aware there are similar questions on SO but most of these questions do not have textScore as sort criteria and therefore my question is different.

Rahat Mahbub Rahat Mahbub · Accepted Answer · 2015-10-19T19:19:41

You can use aggregate to circumvent the limit.

Collection.aggregate([
    { $match: { $text: { $search: searchString } } },
    { $sort: { score: { $meta: "textScore" } } }
])

The $sort stage has a 100 MB limit. If you need more, you can use allowDiskUse, that will write to temp files while sorting takes place. To do that just add allowDiskUse: true to the aggregate option.

If your result is greater than 16MB (i.e. MongoDB's document size limit), you need to request a cursor to iterate through your data. Just add .cursor() before your exec and here's a detailed example. http://mongoosejs.com/docs/api.html#aggregate_Aggregate-cursor

MongoDB fulltext search: Overflow sort stage buffered data usage

1 Answers