Node.js, MongoDB - Inserting/updating multiple documents and sending a single response

Question

I'm trying to develop a synchronization server (think: SVN like) that accepts one or more documents (JSON string) from the client in one request (JSON stringified array of JS objects), inserts/updates them into mongodb and sends one response - which is a JSON string containing the insert/update status (and some more info like _id in mongo) of each document.

If it was one document, i could have done one insert/update and in its callback, i could have sent the response.

collection.insert(_data, function(error, result) {
    if(error) res.send('error');
    else res.send(JSON.stringify({result: result}));
});

But how to do this when i have more than one document. I could do insert/update one document in the previous one's callback. But i fear i'll end up with a dreadful ladder of code if i do that (i could do it in one function and recurse, yes).

Any help will be greatly appreciated. BTW, i'm using this driver: http://mongodb.github.io/node-mongodb-native/

Note: I'm not looking at batch inserts or updates, as each document that is being processed needs individual processing. Some may need to be inserted, some updated, and then there is version number & sync-status checking etc.

mattsch mattsch · Accepted Answer · 2013-09-30T11:39:28

You might want to try the async module for this. It has some very helpful methods for processing each item in a collection, and offers functionality for when all processing has finished.

I refer you in particular to the queue function, which lets you add tasks to a queue, then once all items have been processed, do something.

For instance, you might do something like:

var q = async.queue(function(task, callback) {
  // task.doc would contain your individual doc to process

  // your insert / update logic goes here...

  // Callback signifies you're done with this task
  callback();

}, 2) // <--- this number specifies the number of tasks to run in parallel

q.drain = function() {
  // this is the queue's callback, called when the queue is empty,
  // i.e. when all your documents have been processed.

  res.send(statusCode, message);
}

Then, if we assume you have your list of documents in a variable named docs, all you need to do to process them all is push them onto the queue.

for (var doc in docs) {
  q.push({ doc: docs[doc] }, function(err) {
    if (err) console.log(err);
  })
}

Protip: you need to push an object containing the document onto the queue. There's a weird error if you try to pass in an unwrapped object.

Now, if you wanted the specific statuses for each document you process in Mongo, this is totally possible like this. As long as you instantiated a data structure outside the queue, you could add statusCodes (etc.) to it as each item is processed, and send the structure to the client in the queue's drain function. Shouldn't be too much hassle.

Node.js, MongoDB - Inserting/updating multiple documents and sending a single response

1 Answers