Bulk insert in MongoDB using mongoose

Question

I currently have a collection in Mongodb say "Collection1". I have the following array of objects that need to be into inserted into MongoDB. I am using Mongoose API. For now, I am iterating through the array and inserting each of them into mongo. This is ok for now, but will be a problem when the data is too big. I need a way of inserting the data in bulk into MongoDB without repetition. I am not sure how to do this. I could not find a bulk option in Mongoose.

My code below

myData = [Obj1,Obj2,Obj3.......]

myData.forEach(function(ele){
      //console.log(ele)
     saveToMongo(ele);
    });
function saveToMongo(obj){
    (new Collection1(obj)).save(function (err, response) {
          if (err) {
             // console.log('Error while inserting: ' + obj.name + " " +err);
          } else {
            // console.log('Data successfully inserted');
          }
      });

      return Collection1(obj);
  }

chridam chridam · Accepted Answer · 2016-05-22T20:51:33

You might want to use the insertMany() method here if you're using the latest Mongoose version 4.4.X and greater, which essentially uses Model.collection.insertMany() under the hood and the driver might handle parallelizing >= 1000 docs for you.

myData = [Obj1, Obj2, Obj3.......];
Collection1.insertMany(myData, function(error, docs) {});

or using Promises for better error handling

Collection1.insertMany(myData)
    .then(function(docs) {
         // do something with docs
    })
    .catch(function(err) {
        // error handling here
    });

It works by creating a bunch of documents, calls .validate() on them in parallel, and then calls the underlying driver's insertMany() on the result of toObject({ virtuals: false }); of each doc. Although insertMany() doesn't trigger pre-save hooks, it has better performance because it only makes 1 round-trip to the server rather than 1 for each document.

For Mongoose versions ~3.8.8, ~3.8.22, 4.x which support MongoDB Server >=2.6.x, you could use the Bulk API as follows

var bulk = Collection1.collection.initializeOrderedBulkOp(),
    counter = 0;

myData.forEach(function(doc) {
    bulk.insert(doc);

    counter++;
    if (counter % 500 == 0) {
        bulk.execute(function(err, r) {
           // do something with the result
           bulk = Collection1.collection.initializeOrderedBulkOp();
           counter = 0;
        });
    }
});

// Catch any docs in the queue under or over the 500's
if (counter > 0) {
    bulk.execute(function(err,result) {
       // do something with the result here
    });
}

Bulk insert in MongoDB using mongoose

2 Answers