aws sdk dynamodb scan operation with FilterExpression(contains) is resulting empty items

0
votes

I am trying to fetch data from AWS Dynamo DB scan with contains FilterExpression .I am using the following param but receiving empty response as:

{"Items":[],"Count":0,"ScannedCount":792,"LastEvaluatedKey":{"ID":"b123456789"}}"}

where i want to filter based on string from array ex: orange from fruits:['orange', 'apple', 'banana'];

Param for scan operation:

params = { TableName: table, FilterExpression: "contains (fruits, :L)", ExpressionAttributeValues: { ":L": { S :"orange" } } }

but when i use the same as command of AWS CLI. I am getting data in following format :

{"Items": [{"fruits": {"L": [{"S": "orange"}]}},{}]}

Can you please let me know what is wrong with the param i am using.

1
amazon-dynamodbaws-sdk-nodejsamazon-dynamodb-streams

1 Answers

0
votes

To understand what is happening here, you need to understand how DynamoDB filters data and paginates results. It happens in this order:

  1. Read items from the table
  2. Apply Filter
  3. Return Results

DynamoDB query and scan operations return up to 1MB of data at a time. Anything beyond that will be paginated. You know your results are being paginated if DynamoDB returns a LastEvaluatedKey element in your response.

Filters apply after the 1MB limit. This is the critical step that often catches people off-guard. In your situation, the following is happening:

  1. You execute a scan operation that reads 1MB of data from the table.
  2. You apply a filter to the 1MB response, which results in all of the records in the first step to be eliminated from the response.
  3. DDB returns the remaining items with a LastEvaluatedKey element, which indicates there is more data to search.

In other words, your filter isn't applying to the entire table. It's applying to 1MB of the table at a time. In order to get the results you are looking for, you are going to need to execute the scan operation repeatedly until you reach the last "page" of the table.