DynamoDB scan leaves valid item out

Question

In a DynamoDB table, I have an item with the following scheme:

{
    id: 427,
    type: 'page',
    ...other_data
}

When querying on primary index (id), I get the item returned as expected.

With a scan operation inside AWS DynamoDB web app to get all items with type page, 188 items including this missing item are returned. However, performing this scan operation inside Lambda with the AWS SDK, only 162 items are returned. Part of the code looks like:

const params = { 
    TableName: <my-table-name>,
    FilterExpression: '#type = :type',
    ExpressionAttributeNames: { '#type': 'type' },
    ExpressionAttributeValues: { ':type': 'page' }
};

dynamodb.scan(params, (error, result) => {
    if (error) {
      console.log('error', error);
    } else {
      console.log(result.Items); // 162 items
    }
});

What is missing here?

Have you tried using ConsistentRead set to true? Shouldn't matter if you have tried this multiple times but just to discard you were missing the records because of the reads default eventual consistency — Daniel Conde Marin
@Daniel Unfortunately, that didn't solve it. However, I just found out that implementing stackoverflow.com/a/44590524/5550032 (accepted answer) works for me. Apparently I am hitting some kind of limit. Perhaps it's better to create a GSI from type property. — Kees van Lierop

Daniel Conde Marin Daniel Conde Marin · Accepted Answer · 2017-10-25T08:07:13

This will probably be the case of your result data set exceeding the limit 1MB:

If the total number of scanned items exceeds the maximum data set size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation. The results also include the number of items exceeding the limit. A scan can result in no table data meeting the filter criteria.

Check on the result for the LastEvaluatedKey field and use it for the next scan operation passing it as ExclusiveStartKey

DynamoDB scan leaves valid item out

1 Answers