Google App Engine NDB creating metadata properties

Question

I've been using Google App Engine to develop a prototype inventory application. I've recently switched from using the old DB datastore library to using the new NDB library, and the features are cool, but in making a change to one of my entities, suddenly NDB is inserting a property called "metadata" in my entity. Here's the entity description (the change I made was to add the "inv_posid" and "inv_lastchange" properties, nothing major):

class Inventory(ndb.Model):
    inv_product = ndb.KeyProperty(kind = Product)
    inv_prdcr_name = ndb.StringProperty(default="")
    inv_product_type = ndb.StringProperty(default="")
    inv_product_name = ndb.StringProperty(default="")
    inv_product_year = ndb.IntegerProperty(default=0)
    inv_count = ndb.IntegerProperty(default=0)
    inv_price = ndb.IntegerProperty(default=0)
    inv_glass_price = ndb.IntegerProperty(default=0)
    inv_bin = ndb.StringProperty(default="")
    inv_posid = ndb.StringProperty(default="")
    inv_lastchange = ndb.FloatProperty(default=0.0)

With adding the new properties, I intended to change my query to use the "inv_lastchange" as a filter, and since NDB never includes entities in results that don't have the appropriate property included, I wanted to run a quick sweep through my datastore, to add the properties to all the entities appropriately. So, here's what I did:

...
@ndb.tasklet
def fixInventory(invitem):
    invitem.inv_posid = ""
    invitem.inv_lastchange = 0.0
    invkey = yield invitem.put_async()

inventory = Inventory.query()
output = inventory.map(fixInventory)

I thought it would be neat to play around with tasklets and see how the asynchronous calls worked. However, after doing that, when I went to look in the datastore viewer (on my local datastore), I saw this new "metadata" property, which I just assumed was something NDB needed to have, so I didn't think anything of it.

Until, the next time I tried to update one of my inventory items, I got this error:

  File "/Programming/VirtualCellar/server/virtsom.py", line 2118, in get
    inventory.put()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/model.py", line 3432, in _put
    return self._put_async(**ctx_options).get_result()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 326, in get_result
    self.check_success()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 369, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/context.py", line 810, in put
    key = yield self._put_batcher.add(entity, options)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 369, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/context.py", line 343, in _put_tasklet
    keys = yield self._conn.async_put(options, datastore_entities)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/ndb/tasklets.py", line 455, in _on_rpc_completion
    result = rpc.get_result()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1882, in __put_hook
    self.check_rpc_success(rpc)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1373, in check_rpc_success
    raise _ToDatastoreError(err)
BadRequestError: cannot store entity with reserved property name '__metadata__'

So what's happening? Is there something that I'm forgetting to do here? It feels like the "metadata" property is supposed to be hidden or protected in some way, but it has been added just like a regular property, which is now preventing any other saves to be made to the entity. Anybody run across this before?

I never heard about ndb automatically creating a metadata property. Are you certain the property didn't exist before? How exactly did you change your entity when this property popped up? — Dan Cornilescu

snakecharmerb snakecharmerb · Accepted Answer · 2017-01-11T07:35:53

The __metadata__ property is added to entities when they are saved to the datastore and removed when they are read from the datastore. This is done in the _ToStorageEntity and _FromStorageEntity functions in google/appengine/datastore/datastore_stub_util.py.

Presumably internal failure has corrupted your entities somehow.

You might be able to recover by removing the __metadata__ property from your entity instances' _properties dict and saving.

For example:

for inv in Inventory.query():
    del inv._properties['__metadata__']
    inv.put()

(Maybe backup your local datastore file before trying this, as a precaution against accidents).

Google App Engine NDB creating __metadata__ properties

2 Answers

Google App Engine NDB creating metadata properties