Repeated nested properties of StructuredProperty

Question

While the NDB documentation says:

Although a StructuredProperty can be repeated and a StructuredProperty can contain another StructuredProperty, beware: if one structured property contains another, only one of them can be repeated.

It appeared to me that the clearest interpretation of the documentation may be properly written as "if one structured property contains another structured property".

In which case, I would expect something like this to work:

class KeyList(ndb.Model):
  keys = ndb.KeyProperty(repeated=True)

class Collection(ndb.Model):
  lists = ndb.StructuredProperty(KeyList, repeated=True)

However, this fails with the error:

TypeError: This StructuredProperty cannot use repeated=True because its model class (KeyList) contains repeated properties (directly or indirectly).

It appears that one cannot nest any repeated property inside a repeated StructuredProperty.

From the code, this looks to be the intended behaviour.

In which case the development appserver may differ from the documented behaviour. If the documentation as I've read it is correct, then a fix for the condition in the development server giving rise to the above error might be something like:

if modelclass._has_repeated and isinstance(modelclass, StructuredProperty):
   # ...

Should an AppEngine StructuredProperty be able to contain repeated properties (other than a repeated StructuredProperty)?

Issue report is here: https://github.com/GoogleCloudPlatform/appengine-python-vm-runtime/issues/40

EDIT It's worth noting that the golang docs say this:

Slices of structs are valid, as are structs that contain slices. However, if one struct contains another, then at most one of those can be repeated. This disqualifies recursively defined struct types: any struct T that (directly or indirectly) contains a []T.

The implication I take from this is that repeated non-struct properties on repeated structs are not inherently prohibited at the BigTable level, but rather a side-effect of the Python implementation.

Without further corroboration I would not rely on this implication.

I had the same problem and although the documentation says it's possible It is not. — janscas
Use LocalStructuredProperty instead. You can nest many levels. However be aware that's this propery will be stored as a blob. — janscas
@janscas As a blob, the properties of a LocalStructuredProperty cannot be indexed. — Brian M. Hunt
I know... But it,s the only way to nest models in the same entity — janscas
Documentation seems pretty clear. If you want indexed structiured properties don't nest them. — Tim Hoffman

Patrick Costello Patrick Costello · Accepted Answer · 2016-03-22T17:27:16

This is a bug with our documentation --

With StructuredPropertys, you can only have a single layer of repeated properties.

Some background:

ndb handles StructuredPropertys by exploding properties before writing to Datastore. So for example:

class Inner(ndb.Model):
  a = ndb.StringProperty()

class Outer(ndb.Model):
  inner = ndb.StructuredProperty(Inner, repeated=True)

Then if we write: Outer(inner=[Inner(a="1"), Inner(a="2")]), this actually gets written to the Datastore as:

{
  inner.a = ["1", "2"]
}

However, if instead Inner.a were repeated, we could write something like:

 Outer(inner=[Inner(a=["1", "3"]), Inner(a=["2", "4"])])

This would get written to Datastore like:

{
  inner.a : ["1", "3", "2", "4"]
}

Then when we read this, we have no idea how to parse, as all these are valid:

Outer(inner=[Inner(a=["1", "3"]), Inner(a=["2", "4"])])
Outer(inner=[Inner(a=["1", "3", "2"]), Inner(a=["4"])])
Outer(inner=[Inner(a=["1"]), Inner(a=["3"]), Inner(a=["2"]), Inner(a=["4"])])
Outer(inner=[Inner(a=[]), Inner(a=["1", "3", "2", "4"])])

Note that other libraries (for example Objectify) can avoid this issue by storing both the exploded properties for indexing and an unindexed blob of the entire dataset which can maintain the structure for deserialization.

Repeated nested properties of StructuredProperty

2 Answers