DynamoDB, dynamic atomic update of mapped values with AWS Lambda (NodeJS runtime)

Question

I am trying to figure out how I could perform atomic updates on an item where the source data contains mapped values with the keys of those maps being dynamic.

If you look at the sample data below, I am trying to figure out how I could do atomic updates of the values in BSSentDestIp and BSRecvDestIp over the same item. I was reading the documentation but the only thing I could find was list_append, which would leave me with a list of appended keys/values that I would need to traverse and sum later.

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html

Example of input data:

{
  "RecordId": 31,
  "UUID": "170ae748-f8cf-4df9-6e08-c0c8a5f029d4",
  "UserId": "username",
  "DeviceId": "e0:cb:4e:53:ae:ff",
  "ExpireTime": 1501445446,
  "StartTime": 1501441846,
  "EndTime": 1501441856,
  "MinuteId": 10,
  "PacketCount": 1028,
  "ByteSum": 834111,
  "BSSent": 98035,
  "BSRecv": 736076,
  "BSSentDestIp": {
    "151.101.129.69": 2518,
    "192.168.1.254": 4780,
    "192.168.1.80": 14089,
    "192.33.31.162": 2386,
    "54.239.30.232": 21815,
    "54.239.31.129": 6423,
    "54.239.31.69": 3255,
    "54.239.31.83": 18447,
    "98.138.253.109": 3020
  },
  "BSRecvDestIp": {
    "151.101.129.69": 42414,
    "151.101.57.174": 20792,
    "192.230.66.108": 130175,
    "192.33.31.162": 56398,
    "23.194.140.100": 26209,
    "54.239.26.209": 57210,
    "54.239.31.129": 188747,
    "54.239.31.69": 41115,
    "98.138.253.109": 111775
  }
}

NodeJS function executed via Lambda to update Dynamo:

function updateItem(UserIdValue, MinuteIdValue) {

    var UpdateExpressionString = "set PacketCount = PacketCount + :PacketCount, \
    ByteSum = ByteSum + :ByteSum, \
    BSSent = BSSent + :BSSent, \
    BSRecv = BSRecv + :BSRecv";

    var params = {
        TableName: gDynamoTable,
        Key: {
            "UserId": UserIdValue,
            "MinuteId": MinuteIdValue
        },
        UpdateExpression: UpdateExpressionString,
        ExpressionAttributeValues: {
            ":PacketCount": gRecordObject.PacketCount,
            ":ByteSum": gRecordObject.ByteSum,
            ":BSSent": gRecordObject.BSSent,
            ":BSRecv": gRecordObject.BSRecv
        },
        ReturnValues: "UPDATED_NEW"
    };

    dynamo.updateItem(params, function(err, data) {
        if (err) {
            console.log("updateItem Error: " + err);
        } else {
            console.log("updateItem Success: " + JSON.stringify(data));
        }
    });
}

Ivan Mushketyk Ivan Mushketyk · Accepted Answer · 2017-07-31T11:05:56

Updating a singe item is atomic in DynamoDB if you read an item, and call PutItem it is guaranteed to be atomic. It either update all fields or update none of them.

Now the only issue that I see with that is that you can have write conflicts. Say if one process reads an item, update one map, while another process in parallel does the same thing it will result in one PutItem overwriting recent update and you can lose data.

To solve this issue you can use conditional updates. In a nutshell it allows you to update an item only if a specified condition is met. What you can do is to maintain a version number with every item. When you update an item you can increment a version attribute and, when you write an item, check that version number is the one you expect. Otherwise you need to read the item again (somebody updated it while you were working with it) perform your update again and try to write again.

DynamoDB, dynamic atomic update of mapped values with AWS Lambda (NodeJS runtime)

1 Answers