How index GAE Datastore properties when adding indexed to a property

Question

Other than than re-writing an entity to the datastore, is there a method to get existing entity properties indexed?

A scenario: I created new queries using a property of previously created entities. The queries failed with "Cannot query for un-indexed property ", which is correct behaviour as the initial Class definition had (indexed=false) for the relevant property.

I then set (indexed=true) for the relevant property. The queries using this property than ran without generating errors, the output however was INCORRECT!

I assume this is because existing entities do not automatically get added to the index (although Google documents elude to indexes being automatically generated).

I could get the index updated and queries working correctly only by updating each entity. This is perhaps OK if one is aware of the problem is using small datasets. I am concerned about live data. I need to do more testing with live but the same effect/behaviour could be the same.

It appeared changing index.yaml and restarting the GAE instance had no effect. It also appears gcloud datastore create-indexes index.yaml will not affect this behaviour.

What appears to be a solution It seems the only solution is to write back the existing entities so an entry is created in the new index. Probably to the effect of:

all_entities_in_kind=theModel.query()
list_of_keys= ndb.put_multi(all_entities_in_kind)

If there's a better way please post.

Simulation of the effect To illustrate, I ran the code snippet below from the Dev SDK Interactive console. First run of the code below creates test data. The queries will fail (correct behaviour).

Changing bool_p to indexed=true, re-running the modified code means the query will run but the results are incorrect.

Deleting all the data and re-running the code (with bool_p indexed) the queries return the correct result.

Observation It appears the index on the property is not automatically generated. At least from the Interactive console. Restarting the Instance has no effect. Changing the index.yaml file also seems to make no difference. It appears to be the index on the property needs to be built but so far I have not discovered how. The only solution would be export all the data and re-import with the modified class. This is not so much a problem in development but not for the live datastore.

Code example

from google.appengine.ext import ndb
from datetime import datetime, timedelta

class TModel(ndb.Model):
    test_date = ndb.DateProperty(indexed=True)
    text = ndb.StringProperty(indexed=True)
    bool_p = ndb.BooleanProperty(indexed=False) #First run
    #bool_p = ndb.BooleanProperty(indexed=True) #Second run.

query0=TModel.query()
if query0.count() == 0:
    print 'Create test data'
    TModel(id='a',test_date=datetime.strptime('2017-01-01','%Y-%m-%d').date(),text='One',bool_p=True).put()
    TModel(id='b',test_date=datetime.strptime('2017-01-02','%Y-%m-%d').date(),text='Two',bool_p=False).put()
    TModel(id='c',test_date=datetime.strptime('2017-01-03','%Y-%m-%d').date(),text='Three',bool_p=True).put()
    TModel(id='d',test_date=datetime.strptime('2017-01-01','%Y-%m-%d').date(),text='One').put() #To check behaviour with bool_p undefined

query1 = TModel.query(TModel.bool_p == True)
print query1.count()

query2 = TModel.query(TModel.bool_p == False)
print query2.count()

query3 = TModel.query(TModel.test_date <= datetime.strptime('2017-01-02','%Y-%m-%d').date())
print query3.count()

query4 = TModel.query(TModel.test_date <= datetime.strptime('2017-01-02','%Y-%m-%d').date(), TModel.bool_p == True)
print query4.count()

#Equivalent Queries using GQL
queryG1=ndb.gql('SELECT * FROM TModel WHERE bool_p = True')
print queryG1.count()

queryG2=ndb.gql('SELECT * FROM TModel WHERE bool_p = True')
print queryG2.count()

queryG3 =ndb.gql("SELECT * FROM TModel WHERE test_date <= DATE('2017-01-02')")
print queryG3.count()

queryG4 =ndb.gql("SELECT * FROM TModel WHERE test_date <= DATE('2017-01-02') AND bool_p = True ")
print queryG4.count()

Correct Results/Incorrect Result after changing: (indexed=False) to True

2 0
1 0
3 3
1 0
2 0
2 0
3 3
1 0

Tony P. Tony P. · Accepted Answer · 2017-07-14T15:01:53

It is necessary to rewrite the existing entities when changing a property from unindexed to indexed in order to re-index the properties. From https://cloud.google.com/datastore/docs/concepts/indexes#unindexed_properties:

Note, however, that changing a property from excluded to indexed does not affect any existing entities that may have been created before the change. Queries filtering on the property will not return such existing entities, because the entities weren't written to the query's index when they were created. To make the entities accessible by future queries, you must rewrite them to Cloud Datastore so that they will be entered in the appropriate indexes.

How index GAE Datastore properties when adding indexed to a property

1 Answers