Google Cloud Datastore Cursor with google.cloud.ndb

Question

I am working with Google Cloud Datastore using the latest google.cloud.ndb library I am trying to implement pagination use Cursor using the following code. The same is not fetching the data correctly.

[1] To Fetch Data:

query_01 = MyModel.query()
f = query_01.fetch_page_async(limit=5)

This code works fine and fetches 5 entities from MyModel I want to implementation pagination that can be integrated with a Web frontend

[2] To Fetch Next Set of Data

from google.cloud.ndb._datastore_query import Cursor
nextpage_value = "2"
nextcursor = Cursor(cursor=nextpage_value.encode()) # Converts to bytes
query_01 = MyModel.query()
f = query_01.fetch_page_async(limit=5, start_cursor= nextcursor)

[3] To Fetch Previous Set of Data

previouspage_value = "1"
prevcursor = Cursor(cursor=previouspage_value.encode())
query_01 = MyModel.query()
f = query_01.fetch_page_async(limit=5, start_cursor=prevcursor)

The [2] & [3] sets of code do not fetch paginated data, but returns results same as results of codebase [1].

Please note I'm working with Python 3 and using the latest "google.cloud.ndb" Client library to interact with Datastore I have referred to the following link https://github.com/googleapis/python-ndb

I am new to Google Cloud, and appreciate all the help I can get.

appoll appoll · Accepted Answer · 2020-10-19T17:31:31

Firstly, it seems to me like you are expecting to use the wrong kind of pagination. You are trying to use numeric values, whereas the datastore cursor is providing cursor-based pagination.

Instead of passing in byte-encoded integer values (like 1 or 2), the datastore is expecting tokens that look similar to this: 'CjsSNWoIb3Z5LXRlc3RyKQsSBFVzZXIYgICAgICAgAoMCxIIQ3ljbGVEYXkiCjIwMjAtMTAtMTYMGAAgAA=='

Such a cursor you can obtain from the first call to the fetch_page() method, which returns a tuple:

(results, cursor, more) where results is a list of query results, cursor is a cursor pointing just after the last result returned, and more indicates whether there are (likely) more results after that

Secondly, you should be using fetch_page() instead of fetch_page_async(), since the second method does not return you the cursors you need for pagination. Internally, fetch_page() is calling fetch_page_async() to get your query results.

Thirdly and lastly, I am not entirely sure whether the "previous page" use-case is doable using the datastore-provided pagination. It may be that you need to implement that yourself manually, by storing some of the cursors.

I hope that helps and good luck!

Google Cloud Datastore Cursor with google.cloud.ndb

1 Answers