Read embedded entity from python ndb client

Question

I am using the google cloud datastore python client to write an entity into the datastore which contains an embedded entity. An example entity might look like:

data_type: 1
raw_bytes: <unindexed blob>
values: <indexed embedded entity>

I checked the data from the console and the data is getting saved correctly and the values are present.

Next, I need to run a query from a python app engine application. I have represented the above as the following entity in my app engine code:

class DataValues(ndb.Model):
    param1 = ndb.BooleanProperty()
    param2 = ndb.IntegerProperty()
    param3 = ndb.IntegerProperty()

class MyEntity(ndb.Expando):
    data_type = ndb.IntegerProperty(required=True)
    raw_bytes = ndb.BlobProperty()
    values = ndb.StructuredProperty(DataValues)        

One of the filters in the query depends on a property in values. Sample query code is as below:

MyEntity.query().filter(MyEntity.data_type == 1).filter(MyEntity.values.param1 == True).get()

I have created the corresponding composite index in my index.yaml The query runs successfully but the resulting entity contains the embedded entity values as None. All other property values are present.

What can be the issue here ?

Answer 1

Add properties of DataValues entity as properties of the MyEntity.

Answer 2

This is a bit of a guess, but since datastore attributes are kind of keyed by both their name (in this case values) and the name of the "field type/class" (i.e. StructuredProperty), this might fix your problem:

class EmbeddedProperty(ndb.StructuredProperty):
    pass

class MyEntity(ndb.Expando):
    data_type = ndb.IntegerProperty(required=True)
    raw_bytes = ndb.BlobProperty()
    values = EmbeddedProperty(DataValues) 

Give it a shot and let me know if values starts coming back non-null.

Answer 3

I struggled with the same problem, wanting to convert the embedded entity into a Python dictionary. One possible solution, although not a very elegant one, is to use a GenericProperty:

class MyEntity(ndb.Model):
    data_type = ndb.IntegerProperty(required=True)
    raw_bytes = ndb.BlobProperty()
    values = ndb.GenericProperty()

values will then be read as an "Expando" object: Expando(param1=False,...). You can access the individual values with values.param1, values.param2 etc. I would prefer having a custom model class, but this should do the job.