python - Google App Engine NDB: How to store document structure?

Question

From App Engine NDB documentation:

The NDB API provides persistent storage in a schemaless object datastore. It supports automatic caching, sophisticated queries, and atomic transactions. NDB is well-suited to storing structured data records.

I want to create a structure like the following using NDB, where each instance looks like :

{
 city: 'SFO'
 date: '2013-01-27'
 data: {
           'keyword1': count1,
           'keyword2': count2,
           'keyword3': count3,
           'keyword4': count4,
           'keyword5': count5,
           ....
       }
}

How can I design such a schema-less entity in Google App Engine(GAE) using NDB?
I am new to GAE and not sure how to achieve this

Thank you

Have a look at the NDB docs. The data part you can save in a : JSON property or a structured repeated property or a blobpropert or a pickle propery. — voscausa
@voscausa if he needs to query by custom attribute he can't use those types. — Shay Erlichmen

dting dting · Accepted Answer · 2013-01-29T02:40:17

If you don't need to query for the attributes in data you can use one of the properties as mentioned by @voscausa:

JsonProperty

class MyModel(ndb.Model):
  city = ndb.StringProperty()
  date = ndb.DateProperty()
  data = ndb.JsonProperty()

my_model = MyModel(city="somewhere", 
                   date=datetime.date.today(),
                   data={'keyword1': 3,
                         'keyword2': 5,
                         'keyword3': 1,})

StructuredProperty:

class Data(ndb.Model):
  keyword = ndb.StringProperty()
  count = ndb.IntegerProperty()

class MyModel(ndb.Model):
  city = ndb.StringProperty()
  date = ndb.DateProperty()
  data = ndb.StructuredProperty(Data, repeated=True)

my_model = MyModel(city="somewhere", 
                   date=datetime.date.today(),
                   data=[Data(keyword="keyword1", count=3),
                         Data(keyword="keyword2", count=5),
                         Data(keyword="keyword3", count=1)])
my_model.put()

The problem here is filtering for structured properties. The properties of Keyword are viewed as parallel arrays. Doing a query such as:

q = MyModel.query(MyModel.data.keyword=='keyword1',
                  MyModel.data.count > 4)

would incorrectly include my_model.

https://developers.google.com/appengine/docs/python/ndb/queries#filtering_structured_properties

Using an expando model would work and allow you to query for keywords:

class MyModel(ndb.Expando):
  city = ndb.StringProperty()
  date = ndb.DateProperty()

m = MyModel(city="Somewhere", date=datetime.date.today())
m.keyword1 = 3
m.keyword2 = 5
m.keyword3 = 1
m.put()

q = MyModel.query(ndb.GenericProperty('keyword1') > 2)

https://developers.google.com/appengine/docs/python/ndb/entities#expando

python - Google App Engine NDB: How to store document structure?

2 Answers