NodeJS Express pagination with Google Datastore how to integrate cursor queries with UI control

Question

I am stuck on implementing Pagination and I just need a bit of help, either some example code or even just a hint to help me proceed in the right direction.

I'm looking for some guidance on how to integrate Google Datastore database cursors with front-end UI pagination controls. I know how to build an angular pagination service, but that's with retrieving all the data at once and due to performance issues (5,000 records+) I want to use cursors to retrieve data in subsets.

NOTE: There's a similar question here, but I need more detail than this accepted answer provides. Node pagination in google datastore

QUESTION: How can I integrate the paginated datastore cursor queries with the front-end UI controls to allow the user to select the current page and control number of results displayed on each page?

I need to build a page that displays a large number of records with dynamic pagination. The user must be able to select the number of records display on each page.

Since there are several thousand records that might be returned at one time, I want to use cursors to retrieve subsets of data.

There is an example of how to paginate in the docs, but it is a pretty basic example and does not demonstrate how to integrate with front-end UI controls.

Can anyone provide a more detailed example and/or point me in the right direction on where to begin with this requirement? Unfortunately I haven't been able to find any detailed examples online.

https://googlecloudplatform.github.io/google-cloud-node/#/docs/datastore/1.1.0/datastore

Paginating Records

var express = require('express');
var app = express();

var NUM_RESULTS_PER_PAGE = 15;

app.get('/contacts', function(req, res) {
  var query = datastore.createQuery('Contacts')
    .limit(NUM_RESULTS_PER_PAGE);

  if (req.query.nextPageCursor) {
    query.start(req.query.nextPageCursor);
  }

  datastore.runQuery(query, function(err, entities, info) {
    if (err) {
      // Error handling omitted.
      return;
    }

    // Respond to the front end with the contacts and the cursoring token
    // from the query we just ran.
    var frontEndResponse = {
      contacts: entities
    };

    // Check if  more results may exist.
    if (info.moreResults !== datastore.NO_MORE_RESULTS) {
      frontEndResponse.nextPageCursor = info.endCursor;
    }

    res.render('contacts', frontEndResponse);
  });
});

Dan Cornilescu Dan Cornilescu · Accepted Answer · 2017-11-13T20:33:30

One thing to keep in mind is the first item on the Limitations of cursors list:

A cursor can be used only by the same project that performed the original query, and only to continue the same query. It is not possible to retrieve results using a cursor without setting up the same query from which it was originally generated.

So you need to always be able to re-create the original query inside your handler, which means you need to pass around the equivalent of your NUM_RESULTS_PER_PAGE value from one request to another. You also need to reset the query every time that value changes - meaning you can't continue browsing from where you were after changing the number of results displayed per page.

Then, to be able to use the pagination, you also need to pass around the current cursor value from one request to another, updated at every request.

Now I'm not a NodeJS user, so I can tell exactly how this passing values around from one request to another would typically be implemented. In your code req.query.nextPageCursor and frontEndResponse.nextPageCursor appear to be intended for this, but I can't tell if that's OK or not. Maybe this is a less specific question easier to find an answer for.

In python webapp2, for example, I can store such values server-side in the user's session in one request and read them from the session in a subsequent request. Donno if this is of any help.

NodeJS Express pagination with Google Datastore how to integrate cursor queries with UI control

1 Answers