Concurrency error while executing DocumentDB stored procedure on multiple Docker containers

Question

I am currently building a python Tornado web application using Azure Storage to store images, and DocumentDB to store metadata on the images. Whenever an image is uploaded, it can use any 1 of 2 possible Docker containers running the Tornado Web App to execute the POST method asynchronously. The error I'm having is when I get to the stored procedure I have sitting in DocumentDB scripts. The sproc is being executed in two separate threads in two separate Docker containers at the same time. The stored procedure is meant to generate a new ReceiptID for each image uploaded by querying DocDB for the gen_receipt_id document which looks like this:

{
  "id": "gen_receipt_id",
  "counter": 406
}

The sproc then increments the counter property by 1, and that new ID is used to be attached to the new receipt in metadata. The sproc looks like this:

function receiptIDSproc() {
    var collection = getContext().getCollection();

    // Query documents and take 1st item.
    var isAccepted = collection.queryDocuments(
        collection.getSelfLink(),
        "SELECT * FROM root r WHERE r.id='gen_receipt_id'",
        function(err, feed) {
            if (err) throw err;
            // Check the feed and if empty, set the body to 'no docs found', 
            // else take 1st element from feed
            if (!feed || !feed.length) getContext().getResponse().setBody('no docs found');
            else {
                tryUpdate(feed[0]);
            }
        });

    if (!isAccepted) throw new Error('The query was not accepted by the server.');

    function tryUpdate(document) {

        var requestOptions = {"If-Match": document._etag,
                               etag: document._etag};
        document.counter += 1;
        // Update the document.
        var isAccepted = collection.replaceDocument(document._self, document, requestOptions, function (err, updatedDocument,responseOptions) {
            if (err) throw err;
            // If we have successfully updated the document - return it in the response body.
            getContext().getResponse().setBody(updatedDocument);
        });

        // If we hit execution bounds - throw an exception.
        if (!isAccepted) {
            throw new Error("The stored procedure timed out.");
        }
    }
}

However, when I go to upload multiple images concurrently, I get a conflict with the operation happening asynchronously: Fine-Uploader upload conflict

The error in console looks like this:

[36mtornado2_1  |[0m ERROR:500 POST /v1.0/groups/1/receipts (172.18.0.4) 1684.98ms
[33mtornado1_1  |[0m 407 //Here I'm printing the ID the Sproc generated
[33mtornado1_1  |[0m 2016/9/13/000000000000407
[36mtornado2_1  |[0m 407 //Here I'm printing the ID the Sproc generated
[36mtornado2_1  |[0m 2016/9/13/000000000000407
[32mnginx_1     |[0m 10.0.75.1 - - [13/Sep/2016:16:49:47 +0000] "POST /v1.0/groups/1/receipts HTTP/1.1" 200 17 "http://local.zenxpense.com/upload" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
[33mtornado1_1  |[0m INFO:200 POST /v1.0/groups/1/receipts (172.18.0.4) 1132.49ms
[36mtornado2_1  |[0m WARNING:500 POST /v1.0/groups/1/receipts (172.18.0.4): An error occured while uploading to Azure Storage: HTTP 500: Internal Server Error (An error occured while creating DocumentDB record: Status code: 409
[36mtornado2_1  |[0m {"code":"Conflict","message":"Message: {\"Errors\":[\"Resource with specified id or name already exists\"]}\r\nActivityId: b226be91-f193-4c1b-9cc2-bcd8293bd36b, Request URI: /apps/8ae2ad5a-d261-42ac-aaa1-9ec0fd662d12/services/cc7fdf37-5f62-41db-a9d6-37626da67815/partitions/8063ad6c-33ad-4148-a60f-91c3acbfae6f/replicas/131171655602617741p"})

As you can see from the error, the Sproc is executing and generating the same ReceiptID on two different Docker containers, 407 and because of that, there's a conflict error since I'm trying to create two documents with the same ID. What I need to happen is prevent the Sproc from generating the same ID on two separate containers. I tried using Etags and the "If-Match" header in the Sproc, but it still happens since each container has the same Etag on the document, so it doesn't see an error.

Larry Maccherone Larry Maccherone · Accepted Answer · 2016-09-13T17:58:18

The typical NoSQL solution to this common problem is to use GUIDs rather than sequential IDs.

However, since DocumentDB sprocs provide you with ACID constraints, it should be possible to do what you want using an optimistic concurrency approach with retry.

So, if you run this sproc twice from two different places at the exact same time, you'll get the same ID (407 in this example). One of the sprocs should be able to write to that ID and the other will fail. The key here is to retry any that fail. The sproc will rerun and get the next ID (408). Since simultaneous requests should be a rarity, there should be negligible impact on the median response time.

Concurrency error while executing DocumentDB stored procedure on multiple Docker containers

1 Answers