GAE - Interpreting datastore indexes limits

Question

Google documentation says:

The Datastore imposes limits on the number and overall size of index entries that can be associated with a single entity. These limits are large, and most applications are not affected. However, there are circumstances in which you might encounter the limits.

I have some difficulties to interpret these words (probably beacuse of my ignorance about the argument). By default The datastore creates a single index for any (not defined as unindexed) entity property. Therefore if I define entity PIZZA as:

PIZZA{
   "name" : "somename";
   "price" : <someprice>;
   "property3" : "...";
   ...
   "propertyN" : "...";
}

Then when I "put" this entity in the datastore, it creates N+2 indexes (one for each property defined).

Now I decide to create another PIZZA entity with M properties and R of these M properties are different from the N+2 defined in the first entity.
Thus the database will insert the new entity in the index tables of the M-R properties that are in common with the first PIZZA and creates additional R indexes.

Now the question: the indexes limits are on the number of indexed properties of any enetity (in this case on N+2) or on the total number of indexes that can contain PIZZA entities? (in this case N+R) ? or again on the total number of indexes?

Dan Sanderson Dan Sanderson · Accepted Answer · 2014-09-20T18:06:33

This size limit refers to the number of index fields (rows times properties) to which the entity contributes property values, not a number of indexes. When referring just to the built-in (automatic) indexes, it's simply a count of the indexable properties of an entity, since each property value appears just once in each built-in index. The count increases as that property contributes to custom indexes for complex queries.

This limit is more noticeable when you have indexed multi-valued properties. Such a property contributes one row per value to the built-in index. It also increases the number of times each property appears in a custom index, because all combinations of all indexed values must be represented in the custom index, so many get repeated. If I have a custom index of Pizza entities with a crust_type single-value property and a toppings multi-valued property, with an entity that has three toppings values, the crust_type property contributes three times to that index, one for each toppings value.

There's a related limit to the total value size equal to the size of each value multiplied by the number of index rows in which the value appears. Both limits are for the properties of a single entity.

GAE - Interpreting datastore indexes limits

1 Answers