Google Datastore, "IN" query filter and pagination

Question

We have an application running in Google App Engine and storing data in Google Datastore.

For a given Datastore kind, all our entities have a property type.

We are interested in running a query with an IN query filter to fetch multiple types at once, something like:

type in ['event', 'comment', 'custom']

As there are thousands of entities within this kind, pagination is needed.

The problem we are having is that it is a known limitation of the Datastore that queries with "IN" filters do not support cursor.

Are there sensible ways to get around this limitation?

Using offset would be costy and not performant. Also we can't fetch all entities and filter in the client as we are building an API, hence we don't develop the client ourselves.

Any hint would be really appreciated, thanks!

have a look at this stackoverflow.com/questions/6014657 . The accepted answer would work in most scenarios that are similar. Nick Johnsons answer should also give you some insight into the topic. — konqi
Thanks @konqi I will investigate if we can get around the cursor limitation with a timestamp property in the entities. — achatain

Andrei Volgin Andrei Volgin · Accepted Answer · 2016-02-23T17:43:28

IN filter results in individual EQUAL queries for each item on the list. This is why they do not support cursors - in your case, there will be 3 distinct positions in the index after your run the IN query.

Consider instead adding another property to your entity, which will serve as a flag for this type of API call: its value will be "true" if a type is in ['event', 'comment', 'custom'], or "false" otherwise. Maybe this flag may allow you to make "type" property unindexed - that would be an additional benefit.

With this new indexed property you can use a regular EQUAL filter. It will be faster (1 query instead of 3), and you can use cursors for pagination.

Google Datastore, "IN" query filter and pagination

1 Answers