AWS DynamoDB Backwards Scanning Ignores LastEvaluatedKey

Question

I am using AWS for hosting an API with API Gateway and DynamoDB direct integration.

I am trying now to add the pagination feature for my app, and I am having a hard time to implement it 100%. The problem that I'm facing is when I scan backwards, and I'll give you an example to understand the problem better.

Imagine that I have a list of 20 items and a page size 5.

1. When I query my table, the first query will return 1, 2, 3, 4, 5. Expected behavior!
2. When I query my table, passing it the lastEvaluatedKey from the fifth element, it returns 6, 7, 8, 9, 10. Expected behavior.
3. Now, my problem stands when I query using the flag ScanIndexForward=false. This means that I don't want to move forward. Instead, I want the previous items from the lastEvaluatedKey. The thing is, if I use the lastEvaluatedKey retrieved from my last query, instead of having something like 5, 4, 3, 2, 1, I have 4, 3, 2, 1.

The element from the lastEvaluatedKey or call it head if you want, is skipped.

If you imagine a normal behavior inside a table, where you're clicking next page and previous page, that would mean that when you go to the second page, and come back to the first page, you'll get only 4 items, instead of one, for the proposed scenario.

Finally, I have tried the same query with the AWS CLI and the result is the same. After having a chat with the Support guys, they confirmed that this is the expected behavior.

I just want to know how people treat this scenario, since I know that Amazon uses it in production, so there should be a way of doing that.

Answer 1

To understand the behavior you saw, you need to understand that the original purpose of LastEvaluatedKey was not to do random-access to some place in the middle of a query in the way you wanted to use it - but rather resume a query after it got stopped at a page boundary. With that in mind, when ScanIndexForward=false it assumes it is continuing an ongoing reverse query; So if you pass LastEvaluatedKey=5 it assumes the previously returned sequence was 10, 9, 8, 7, 6, 5 (the last item returned was 5) - so will now continue to 4, 3, 2, 1. Exactly as you noted.

If for your purposes you need the "5" item as well, you can just read this item separately in a separate request. It won't cost you more, because Amazon bills you by read item size - it's not cheaper when multiple items are returned in the same query. Also latency won't increase if you do the read and the query in parallel.

Update:

As Charles noted in a comment below, for short items and short pages, an additional GetItem request for "5" will actually cost you with another RCU which may be significant, so it is not a great idea.

So there is another option, which is not to use LastEvaluatedKey at all. Instead, Query allows you to specify in KeyConditionExpression not just the desired partition key, but also a range of sort keys. You can specify the range "sort <= 5" and this will fetch to you the last (when ScanIndexForward=false) page of items for sort <= 5 - including 5. I think this is indeed an even better solution than what I proposed above, because it doesn't waste an additional RCU.