DynamoDB query/scan only returns subset of items

Question

I noticed that DynamoDB query/scan only returns documents that contain a subset of the document, just the key columns it appears.

This means I need to do a separate Batch_Get to get the actual documents referenced by those keys.

I am not using a projection expression, and according to the documentation this means the whole item should be returned.1

How do I get query to return the entire document so I don't have to do a separate batch get?

One example bit of code that shows this is below. It prints out found documents, yet they contain only the primary key, the secondary key, and the sort key.

t1 = db.Table(tname)

q = { 
    'IndexName': 'mysGSI',
    'KeyConditionExpression': "secKey= :val1 AND " \
                              "begins_with(sortKey,:status)",
    'ExpressionAttributeValues': {
        ":val1": 'XXX',
        ":status": 'active-',
    }
}
res = t1.query(**q)

for doc in res['Items']:
    print(json.dumps(doc))

No, this is not right. A simple awscli query to a DynamoDB table will show you all the attributes of the requested item. Can we see some code? — jarmod
@jarmod I added some code I used to test this. Other programs show the same behavior though. — aaa90210
@jarmod I believe so yes, but that doesn't stop batch_get_items returning the whole doc. — aaa90210
BatchGetItem doesn't use an index so it's not limited to the projected attributes of any given index. — jarmod

jarmod jarmod · Accepted Answer · 2019-05-21T01:19:24

This situation is discussed in the documentation for the Select parameter. You have to read quite a lot to find this, which is not ideal.

If you query or scan a global secondary index, you can only request attributes that are projected into the index. Global secondary index queries cannot fetch attributes from the parent table.

Basically:

If you query the parent table then you get all attributes by default.
If you query an LSI then you get all attributes by default - they're retrieved from the projection in the LSI if all attributes are projected into the index (so that costs nothing extra) or from the base table otherwise (which will cost you more reads).
If you query or scan a GSI, you can only request attributes that are projected into the index. GSI queries cannot fetch attributes from the parent table.

DynamoDB query/scan only returns subset of items

1 Answers