Missing index on specific entities in app engine

Question

I have entities in app engine which I query as:

foo = Foo.all().filter('bar =', baz).get()
#baz is unicode, bar is a StringProperty
#Foo inherits from db.Model

This works for most entities, but for some value of baz, no entity is returned, even though the entity certainly exists, as can be verified at https://console.cloud.google.com/datastore/entities/ The cause is that for that specific entity there is no index on it's value of bar, as evidenced by the lack of a checkmark in the 'Indexed' column at that web page.

The docs state that

Indexes for simple queries, such as queries over a single property, are created automatically

So I would have expected that all entities of that type would have an index on that property, but evidently that is incorrect. Questions:

Q1: when the index is created, is it added to entities that were put prior to the first time a query is run using that index? (or is the index created the first time any entity of that type is put?)

Q2: if not, what changes to the entity (if any) will cause the index to be added to that property? (i tried changing a property other than bar, and putting, and that did not cause the entity to be added)

Q3: would explicitly listing the index in index.yaml change this behavior?

Q4: is there a way to programatically determine whether an entity has an index on a specific property?

Q5: (bonus) is there any google documentation on the above?

thanks

Dan McGrath Dan McGrath · Accepted Answer · 2016-05-13T18:55:25

Q1) The index for individual properties is created automatically created when you write the first entity that has that property (with indexed=true). However, whether or not a property is added to the index is an entity/property level attribute that is set when you write it.

Q2) Every property there is a flag that tells the back-end if it should index the property.If you read the entity and write it back down with the flag set to true on bar it will be inserted into the index.

Q3) index.yaml is only for composite indexes (multi-property indexes). Individual properties are controlled by a property-level flag when you write/update the entity and do not need to be pre-configured.

Q4) Only by reading back every entity and checking the index flag for the property in question.

Q5) For composite indexes you can read the Datastore Indexes. For property indexes, read the Entities, Properties, and Keys page down at the "Property and Value Types" section - you'll see lots about indexes there.

Missing index on specific entities in app engine

2 Answers