Avoid mongodb bulk insert duplicate key error

Question

How can I execute a bulk insert and continue in case of duplicate key error?

I have a collection with an unique index on the id field (not _id) and some data in it. Then I get more data and I want to add only the non-present documents to the collection.

I have the following code:

let opts = {
  continueOnError: true, // Neither
  ContinueOnError: true, // of
  keepGoing: true,       // this
  KeepGoing: true,       // works
};
let bulk = collection.initializeUnorderedBulkOp( opts );
bulk.insert( d1 );
bulk.insert( d2 );
bulk.insert( d3 );
...
bulk.insert( dN );
let result = yield bulk.execute( opts ); // this keep throwing duplicate key error

And I just want to ignore the errors and let the bulk finish with all the queued operations.

I searched in npm module api and in the MongoDB api for Bulk, initializeUnorderedBulkOp and the docs for Bulk write with no luck.

Also in the docs for Unordered Operations they say:

Error Handling

If an error occurs during the processing of one of the write operations, MongoDB will continue to process remaining write operations in the list.

Which is not true (at least in my case)

It is misleading and the error case has actually changed in recent releases, as previously an UnOrderdedBukOp construct would never produce an error that "thows", but only produce a "list of errors" in the response. Not the first to complain about this. The general advice is "ignore" the error and inspect the result object yourself, as the result will always continue to the end of the batch anyway. So your are not correct as it actually did write all operations in the list ( that worked ), but it just threw an error, when I think it should not have. — Blakes Seven
Thank you for the clarifications, but my problem is that the execute method, using promises, raises an exception and the promise will be rejected only with the error; losing the BulkWriteResult object. — Volox
Yes. This is exactly my response. Prior drivers did not raise the exception in this case, now they do. This however has no effect on how "UnOrdered" ops are processed. The batch is "still" executed as a whole, however, rather than just returning "errors" in the response object the result "throws" and error as at least one error occured in execution. Understand now? — Blakes Seven
You could simply use bulk.find().upsert().replaceOne() instead of insert. This way, if a document with said id is found, it will be replaced with the new document, otherwise it will be created. No duplicate key errors, consistent state. — Markus W Mahlberg

Stanislav Prusac Stanislav Prusac · Accepted Answer · 2016-12-13T05:05:10

You can use db.collection.insertMany(), (new in version 3.2.) with:

ordered:false

With ordered to false, and in case of duplicate key error, the insert operation would continue with any remaining documents.

Here is link to documentation: https://docs.mongodb.com/v3.2/reference/method/db.collection.insertMany/

Avoid mongodb bulk insert duplicate key error

2 Answers

Ordered Insert in MongoDB

Unordered Insert in MongoDB