Google App Engine ndb Simple Query gives NeedIndexError

Question

I have a Google App Engine project that uses python 2.7, and Cloud Datastore (well it may be Cloud Firestore but more on that later). My issue is that when I do a simple query e.g. Data.query(ndb.AND(Data.timeStamp >= day, Data.timeStamp < day + datetime.timedelta(days=1))).order(Data.timeStamp) I get a 500 (server error) and this error in the logs:

NeedIndexError: no matching index found.
The suggested index for this query is:
- kind: Data
properties:
- name: timeStamp

That suggested index is an index for a simple query. If I go add that query to the index.yaml file and run gcloud datastore indexes create index.yaml, I get this message:

ERROR: (gcloud.datastore.indexes.create) Server responded with code [400]:
Bad Request Unexpected HTTP status 400.
Creating a composite index failed for entity_type: "Data"
Property {
name: "timeStamp"
direction: ASCENDING
}
ancestor: false
: This index: IndexDef{semantics=DATASTORE, kind=Data, indexAncestor=NONE, propertyDefs=[PropertyDef{path=timeStamp, mode=ORDERED, direction=ASCENDING}]}
is not necessary, since single-property indexes are built in.

So I cannot run this query because I need an index, but I am not allowed to make the index. So my question is, how do I run this query?

More info:
Not sure but this may be relevant: When I go into the Google Cloud Console (console.cloud.google.com) and click on the Datastore tab it claims that I use "Cloud Firestore in Native mode" and gives me a link to the Firestore tab. When I go there I can see all my data in Firestore. All this happens even though I use python2 and the ndb api to access Cloud Datastore/Firestore.
Also, when I click on the index page in the Firestore tab, GCP claims that all of my fields are indexed for simple queries.
Finally, the dev_appserver.py did not generate the index for me in the index.yaml file (as expected).

Here is my code:
main.py

import datetime
from google.appengine.ext import ndb

class Data(ndb.Model):
    timeStamp = ndb.DateTimeProperty(indexed=True)


#WSGI compatible function
def app(env, startResponse):

    headers = [('Content-type', 'text/plain')]
    status = "200 OK"

    path = env["PATH_INFO"]

    if path == "/doData":

        Data(timeStamp = datetime.datetime.now()).put()

        startResponse(status, headers)
        return ["done"]


    elif path == "/getData":

        day = datetime.datetime.strptime("2019-06-28", "%Y-%m-%d")

        records = Data.query(ndb.AND(Data.timeStamp >= day, Data.timeStamp < day + datetime.timedelta(days=1))).order(Data.timeStamp).fetch(10)
        print(records)

        startResponse(status, headers)
        return [str(record.timeStamp) + "\n" for record in records]


    startResponse("404 NOT FOUND", headers)
    return ["404 Page Not Found"]

app.yaml:

runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /.*
  script: main.app

index.yaml:

indexes:

- kind: Data
  properties:
  - name: timeStamp

# AUTOGENERATED

# This index.yaml is automatically updated whenever the dev_appserver
# detects that a new type of query is run.  If you want to manage the
# index.yaml file manually, remove the above marker line (the line
# saying "# AUTOGENERATED").  If you want to manage some indexes
# manually, move them above the marker line.  The index.yaml file is
# automatically uploaded to the admin console when you next deploy
# your application using appcfg.py.

Thanks in advance!

Juan Lara Juan Lara · Accepted Answer · 2019-06-28T05:24:29

The extra info is indeed relevant. Your project uses Cloud Firestore in Native mode and not Cloud Firestore in Datastore mode:

When I go into the Google Cloud Console and click on the Datastore tab it claims that I use "Cloud Firestore in Native mode" and gives me a link to the Firestore tab.

If you want to use NDB and the Cloud Datastore API, you'll need to create a new project and select Cloud Firestore in Datastore mode when you create your database.

If you continue using this project, you should use one of the Cloud Firestore client libraries instead.

Google App Engine ndb Simple Query gives NeedIndexError

1 Answers