DynamoDB consistent reads for Global Secondary Index

Question

Why cant I get consistent reads for global-secondary-indexes?

I have the following setup:

The table: tblUsers (id as hash)

Global Secondary Index: tblUsersEmailIndex (email as hash, id as attribute)

Global Secondary Index: tblUsersUsernameIndex (username as hash, id as attribute)

I query the indexes to check if a given email or username is present, so I dont create a duplicate user.

Now, the problem is I cant do consistent reads for queries on the indexes. But why not? This is one of the few occasions I actually need up-to-date data.

According to AWS documentation:

Queries on global secondary indexes support eventual consistency only.

Changes to the table data are propagated to the global secondary indexes within a fraction of a second, under normal conditions. However, in some unlikely failure scenarios, longer propagation delays might occur. Because of this, your applications need to anticipate and handle situations where a query on a global secondary index returns results that are not up-to-date.

But how do i handle this situation? How can I make sure that a given email or username is not already present in the db?

Answer 1

You probably already went through this: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html

The short answer is that you cannot do what you want to do with Global Secondary Indexes (ie it's always eventual consistency).

A solution here would be to have a separate table w/ the attribute you're interested in as a key and do consistent reads there. You would need to ensure you are updating that whenever you are inserting new entities, and you would also have to worry about the edge case in which inserts there succeed, but not in the main table (ie you need to ensure they are in sync)

Another solution would be to scan the whole table, but that would probably be overkill if the table is large.

Why do you care if somebody creates 2 accounts with the same email? You could just use the username as the primary hash key and just not enforce the email uniqueness.

Answer 2

When you try to use putItem, you have a ConditionExpression to use to check if the condition is satisfied to put the item, which means you can check if the email or username exists.

ConditionExpression — (String)
A condition that must be satisfied in order for a conditional PutItem operation to succeed.

An expression can contain any of the following:

Functions: attribute_exists | attribute_not_exists | attribute_type | contains | begins_with | size
These function names are case-sensitive.

Comparison operators: = | <> | < | > | <= | >= | BETWEEN | IN
Logical operators: AND | OR | NOT
For more information on condition expressions, see Condition Expressions in the Amazon DynamoDB Developer Guide.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#putItem-property

Answer 3

Depending on your situation and considering all of the alternatives, it may be acceptable to add an automatic retry when you don't find anything on the GSI the first time to work around the lack of strongly consistent reads. I didn't even think of this until I hit road blocks with other options and then realized this was simple and didn't cause any issues for our particular use case.

{
"TableName": "tokens",

"ProvisionedThroughput": { "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 },

"AttributeDefinitions": [
    { "AttributeName": "data", "AttributeType": "S" },
    { "AttributeName": "type", "AttributeType": "S" },
    { "AttributeName": "token", "AttributeType": "S" }
],

"KeySchema": [
    { "AttributeName": "data", "KeyType": "HASH" },
    { "AttributeName": "type", "KeyType": "RANGE" }
],

"GlobalSecondaryIndexes": [
    {
        "IndexName": "tokens-token",

        "KeySchema": [
            { "AttributeName": "token", "KeyType": "HASH" }
        ],

        "Projection": {
            "ProjectionType": "ALL"
        },

        "ProvisionedThroughput": { "ReadCapacityUnits": 2, "WriteCapacityUnits": 2 }
    }
],

"SSESpecification":  {"Enabled": true }

}

    public async getByToken(token: string): Promise<TokenResponse> {
    let tokenResponse: TokenResponse;
    let tries = 1;
    while (tries <= 2) { // Can't perform strongly consistent read on GSI so we have to do this to insure the token doesn't exist
        let item = await this.getItemByToken(token);
        if (item) return new TokenResponse(item);
        if (tries == 1) await this.sleep(1000);
        tries++;
    }
    return tokenResponse;
}

Since we don't care about performance for someone sending in a non-existent token (which should never happen anyway), we work around the problem without taking any performance hit (other than a possible 1 second delay one time after the token is created). If you just created the token, you wouldn't need to resolve it back to the data you just passed in. But if you happen to do that, we handle it transparently.

Answer 4

I ran across this recently and wanted to share an update. In 2018, DynamoDB added transactions. If you really need to keep two items (either in the same or different tables) in 100% sync with no eventual consistency to worry about, TransactWriteItems and TransactGetItems are what you need.

It's better to avoid the transaction altogether, if you can, as others have suggested.