DynamoDB queryPage operation with FilterExpression returning empty result as well as lastEvaluatedKey

5
votes

As per my understanding. A query operation will seek results on the mentioned index until one of the following condition is met

  • The result set is exhausted.
  • The number of items retrieved reaches the value of the Limit parameter, if specified.
  • The amount of data retrieved reaches the maximum result set size limit of 1 MB. Documented Here

So DynamoDB query will fetch results as per above criteria and then it will apply the FilterExpression so it is quite possible that it might not return any results to you, so it will return empty set and a LastEvaluatedKey

But I also read the following in the documentation

Unlike a Scan operation, a Query operation never returns both an empty result set and a LastEvaluatedKey value.

Can someone please help in explaining what does the above documentation statement actually mean?

Because in practice when I user queryPage API with limit and FilterExpression I am getting opposite of it, i.e. I am getting an empty set as well as LastEvaluatedKey.

Is my above understanding correct? It is possible to get both an empty results and lastEvaluatedKey value? Or I am missing something because of which I am getting empty results? (as per documentation I should not get it. It would be great if I don't get empty results)

2
paginationamazon-dynamodb
You might get key but are you getting value too in lastEvaluatedKey? – Harshal Bulsara
Yes. I am getting values. – manyu
From here filters are applied to the results of the query by dynamo, so perhaps you query up to your limit of responses and then your filter wipes them out? Attempting the same query without the filter should let you know. – jeffrey

2 Answers

1
votes

I am also struggling with same problems as you, after reading the document, I find the following description explaining why. According to Amazon document, "In a Query operation, DynamoDB retrieves the items in sorted order, and then processes the items using KeyConditionExpression and any FilterExpression that might be present."

So it explained why you are getting those results. The filter operation is processed after the query results, so no mater you apply FilterExpression or not, the Query operations are no difference, that's why you see the lastEvaluatedKey is presented, and it's no difference with the one without apply FilterExpression.

So the only possible way to achive what you want is using Global Secondary Indexes, the column you want to filter, you can put it as sort key. In this case, you can query withRangeKeyCondition().

0
votes

They have updated the documentation, and its inline with what your experience is. From the documentation link in the question : Note: A Query operation can return an empty result set and a LastEvaluatedKey if all the items read for the page of results are filtered out.