Fetch multiple documents in a stored procedure (Azure DocumentDB)

Question

I have two document types, Listing and Products. A Listing object contains a list of Products for certain countries, like this:

Listing:

{
  "Name": "Default",
  "Countries": {
    "_default": [
      "4QlxAPFcCAAPAAAAAAAAAA==",
      "4QlxAPFcCAAHAAAAAAAAAA=="
    ],
    "US": [
      "4QlxAPFcCAAIAAAAAAAAAA==",
      "4QlxAPFcCAAHAAAAAAAAAA=="
    ]
  },
  "Type": "Listing",
  "id": "dfed1839-07c5-482b-81c5-669b1dbcd0b6",
  "_rid": "4QlxAPFcCAAEAAAAAAAAAA=="
}

Product:

{
    "Name": "Widget",
    "Price": 3.45,
    "Type": "Product",
    "_rid": "4QlxAPFcCAAHAAAAAAAAAA=="
}

My goal was to create a stored procedure in the Azure DocumentDB collection taking two parameters, ridand country, which would essentially fetch the Listing document, and the documents for that country, in the most efficient manner possible. My presumption is that loading a Document by its resource Id using getContext().getCollection().readDocument(...) would be the fastest way, thus attempting to create a stored procedure for this.

My attempts have been to nest the consecutive calls (callback hell?), using generator/iterators with yield and then with a pure Promise approach. All of the attempts have given the same result:

It will fetch the first document, but will end quite abruptly after the document has been received.

For reference, here's my latest attempt:

function test(rid, country) {
    var collection = getContext().getCollection();
    var collectionSelfLink = collection.getSelfLink();
    var docsLink = collectionSelfLink + "docs/";
    var body = getContext().getResponse().setBody;  

    function getDocument(rid) {
        return new Promise(function(resolve, reject) {
            var accepted = collection.readDocument(docsLink + rid, (err, doc, opts) => {
                resolve(doc);
            });

            if (!accepted)
                reject("Not accepted");            
        });
    }

    getDocument(rid)
        .then(doc => { 
            body("0. First step"); // set test body 

            // Countries is a Dictionary<string, string[]> with resource ids
            return doc.Countries[country] || doc.Countries["_default"];
        })
        // This is how far it gets, resulting in response "1. Documents to fetch: 2"
        .then(a => body("1. Documents to fetch: " + a.length))
        .then(a => a.map(function(productId) { return getDoument(productId); }))
        .then(a => body("2. It should come this far, right?"))
        .then(a => Promise.all(a))
        .then(a => body(a))
        .catch(function(e) { throw new Error(JSON.stringify(e)); });
}

I'm not a promises expert. I tend to hand code my callbacks or use async.js (yes, you can load it as a dependency inside a sproc), but I wonder if your problem is in the line that calls a.map. The wrapping .then will return a promise, which is an array of promises from that point forward. Also, you have a preference for single line functions in your .then() calls but some of those things (calls to body and even call to .map could be done in line. The only thing you really need a promise for is the async call to readDocument. On the other hand this may just be a promise coding style. — Larry Maccherone
I tried the above call chain in a REPL with a mocked getDocument function, and it works. I have also, as noted, tried doing for loop inside an initial collection.readDocument(...), and when using a for-of/for-in/for it will terminate half-way-through (taking 2 out of 4 elements I had in the list). When using a .forEach, it went through the entire array, but didn't wait until the calls were done. I'm curious on how the caller determines when to 'terminate'... — Andre Andersen
How big is the array? How does a fanned out (a.map()) promise chain decide how much parallelization to implement? My new theory is that with a for loop or your a.map promises approach, you fan out too wide and exceed some resource. You may need to limit the parallelization like async.js' ...Limit() methods. — Larry Maccherone
It's been a couple of days with no answer here. What's the current status? — Larry Maccherone
I've posted a proposed answer to my question, as the Promise just did not work whatsoever. @LarryMaccherone, the amount of documents that was fetched was merely between 2 and 4, so the degree of parallelism wasn't in any means high. — Andre Andersen

Andre Andersen Andre Andersen · Accepted Answer · 2017-02-21T05:53:36

It turns out that nesting the calls do in fact work, if you alter the response body frequently(?)

The following procedure worked as expected:

function test(rid, country) {
    var collection = getContext().getCollection();
    var collectionSelfLink = collection.getSelfLink();
    var docsLink = collectionSelfLink + "docs/";
    var body = getContext().getResponse().setBody;

    var accepted = collection.readDocument(docsLink + rid, (err, doc, opts) => {
        if (err) throw new Error(err.message);

        // Countries is a Dictionary<string, string[]> with resource ids
        var offerIds = doc.Countries[country] || doc.Countries["_default"];
        var result = [];

        for (var docId of offerIds) {
            var subAccepted =
                collection.readDocument(docsLink + docId, (err, doc, opts) => {
                    if (err) throw new Error(err.message);

                    result.push(doc);
                });

            if (!subAccepted)
                throw new Error("A subsequent request was not accepted");

            body(result); // <-- Note, setting body in each iteration.
        }
    });

    if (!accepted)
        throw new Error("The request was not accepted");
}

Fetch multiple documents in a stored procedure (Azure DocumentDB)

1 Answers