GAE Cursors not working on queries using IN, what's the workaround?

Question

I'm trying to execute a fairly complex query against GAE's datastore, basically, a subscriber can subscribe to news from a specific town or any town (*), a specific country or any country (*), specific ... or any ... (*)

Now when I want to notify subscribers about news from ZA, I need to find all subscribers that matches country=ZA as well as those matching country=* and the same for the other fields.

Query<Subscriber> query = ofy().load().type(Subscriber.class);

    query  = query.filter("searchCategory IN", Arrays.asList(new String[]{"*", category}));
    query = query.filter("searchCity IN", Arrays.asList(new String[]{"*", city}));
    query = query.filter("searchSuburb IN", Arrays.asList(new String[]{"*", suburb}));
    query = query.filter("searchTown IN", Arrays.asList(new String[]{"*", town}));
    query = query.filter("searchProvince IN", Arrays.asList(new String[]{"*", province}));
    query = query.filter("searchCountry IN", Arrays.asList(new String[]{"*", country}));
    query = query.limit(100);

    QueryResultIterator<Subscriber> iterator = query.iterator();
    while (iterator.hasNext()) {
        Subscriber subscriber = iterator.next();
        System.out.println(iterator.getCursor().toWebSafeString()); // exception here
    }

I'm using task queues to notify enormous amounts of subscribers at a time, 100 at a time (anywhere between 20k and 2M results depending on the query) and cursors seemed like the logical way to break down the massive number of results in manageable chunks ... until I ran the application and got a Null Pointer Exception attempting to get the cursor - turned out that cursors aren't supported when using the IN statement.

Because the NOT_EQUAL and IN operators are implemented with multiple queries, queries that use them do not support cursors, nor do composite queries constructed with the CompositeFilterOperator.or method.

What is the alternative in this situation seeing that I can't use cursors?

Im facing a similar issue, only in my query there is an importance to the order of results. Im breaking my head thinking of a way i can use cursors with 'IN' query and 'ORDER BY'. If you can think of something i would be happy to hear. Thanks. — AsafK
I solved my problem by generating all the permutations of the query and throwing each of them on a task queue as a separate query. I don't think that'll work for sorting unless each task queue inserts that data into a new "table" and does a simple sort and select on that data after all the tasks have completed - very messy! The search API might be a better option as Patrick have mentioned below. — Jan Vladimir Mostert

Omair Shamshir Omair Shamshir · Accepted Answer · 2013-11-12T07:36:28

I think you can use the cursors , according to the documentation

Some NDB queries don't support query cursors, but you can fix them. If a query uses IN, OR, or !=, then the query results won't work with cursors unless ordered by key. If an application doesn't order the results by key and calls fetch_page(), it gets a BadArgumentError. If User.query(User.name.IN(['Joe', 'Jane'])).order(User.name).fetch_page(N) gets an error, change it to User.query(User.name.IN(['Joe', 'Jane'])).order(User.name, User.key).fetch_page(N)

and i suggest you to switch to NDB as its faster and have automatic caching

GAE Cursors not working on queries using IN, what's the workaround?

2 Answers