Why is DynamoDB LastEvaluatedKey depends on the limit of the query?

Question

Using javascript aws-sdk and querying a dynamodb table, with document client. the table contains 10 elements and the query limit = 5.

For each request I use the LastEvaluatedKey to build a new query and send a fresh request, here are the results :

first request --> {Items: Array(5), Count: 5, ScannedCount: 5, LastEvaluatedKey: {…}}
second request --> {Items: Array(5), Count: 5, ScannedCount: 5, LastEvaluatedKey: {…}}
third request --> {Items: Array(0), Count: 0, ScannedCount: 0}

According to this doc

If the result contains a LastEvaluatedKey element, proceed to step 2. If there is not a LastEvaluatedKey in the result, then there are no more items to be retrieved

It is supposed to not return LastEvaluatedKey in the second request, cause there is no more elements, but it returns one which send to an empty result in the third request.

When i try with limit = 4, every things works as expected

first request --> {Items: Array(4), Count: 4, ScannedCount: 4, LastEvaluatedKey: {…}}
second request --> {Items: Array(4), Count: 4, ScannedCount: 4, LastEvaluatedKey: {…}}
third request --> {Items: Array(2), Count: 2, ScannedCount: 2} <--- there is no LastEvaluatedKey as expected

So what is happening here ?

If you don't get an answer here, I suggest raising this on the AWS forums. — F_SO_K
I assume when aws hits the limit '5' it just records the key and since it doesn't look further ahead it doesn't know how much items left to scan. Fairly a simple logic — Can Sahin
@CanSahin this is what i want to know, if it is " according to the doc " an expected result, or not. I would have liked a sourced comment. — sidali

Michael - sqlbot Michael - sqlbot · Accepted Answer · 2018-08-05T12:44:49

What you are seeing is expected.

This statement is true:

If there is not a LastEvaluatedKey in the result, then there are no more items to be retrieved

However, you appear to be assuming that it implies the inverse condition is also true, but it isn't...

If there are no more items to be retrieved, then there is not a LastEvaluatedKey in the result #false

This isn't implied by the first statement, it isn't logically equivalent, and isn't true.

If the service hits your limit exactly, then it does not know whether there may be more matching records to find, so it gives you the last record evaluated so that you can continue the search on the next request. There may be more, there may not be. This is an inevitable consequence of the optimization necessary to get the results back to you as quickly as possible. Otherwise, the service would need to continue looking for at least one more match, even though it has already found enough to satisfy your limit.

If the search reaches the end without also reaching your limit, it knows there's nothing more to search for, so it does not return a LastEvaluatedKey.

Why is DynamoDB LastEvaluatedKey depends on the limit of the query?

2 Answers