GAE NBD entities still seen in Datastore viewer after they are deleted from Python and Memchach is flushed

2
votes

Below is a minimal Python example showing adding entities to NDB, and then deleting them all.

However, though the entities seem to be deleted, the Datastore Viewer still shows them - even after I flush Memcache.

What should I change so that the NDB entities will be deleted?

My code:

from google.appengine.ext import ndb
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

from models import Data

KEY = "fastsimon"
datum_key = dict()

class InvalidHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Invalid entry')

class GetHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Get!\n')
        name = self.request.get('name')
        out_str = "Should not be seen"
        try:
            ancestor_key = datum_key[name]
            qry = Data.owner_query(ancestor_key)
            data = qry.fetch()
            out_str = data[-1].value
        except KeyError:
            out_str = "None"
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write(out_str)

class EndHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('End!')
        qry = Data.query().iter(keys_only=True)
        print "before delete"
        list_of_keys = list()
        for q in qry:
            print "ndb.Key('Datum',q):",ndb.Key('Datum',q.integer_id())
            list_of_keys.append(ndb.Key('Datum',q.integer_id()))
        print "list_of_keys:",list_of_keys
        ndb.delete_multi(list_of_keys)
        print "after delete"
        for q in qry:
            print "ndb.Key('Datum',q):",ndb.Key('Datum',q.integer_id())

class SetHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Set!\n')
        name=self.request.get('name')
        data = Data(parent=ndb.Key("Datum", KEY),
                    name=name,
                    value=self.request.get('value'))
        print "data:",data
        ret_key = data.put()
        datum_key[name] = ret_key

def main():
    application = webapp.WSGIApplication([
            ('/get', GetHandler),
            ('/set', SetHandler),
            ('/end', EndHandler)],     
            debug=False) 
    global app
    app = application
    run_wsgi_app(app)

if __name__ == 'main':
    main()

models.py:

from google.appengine.ext import ndb

class Data(ndb.Model):
    name = ndb.StringProperty() # Upto 500 characters
    value = ndb.TextProperty(required=True) # Unlimited length

    @classmethod
    def owner_query(cls, parent_key):
        return cls.query(ancestor=parent_key).order(cls.name)

The steps I took:

  1. Cleared Datastore:

Cleared Datastore

  1. Executed the following in the browser:
  2. http://localhost:8080/set?name=a&value=1
  3. http://localhost:8080/set?name=b&value=2
  4. http://localhost:8080/end

The Debugging lines are:

data: Data(key=Key('Datum', 'fastsimon', 'Data', None), name=u'a', value=u'1')
INFO     2016-10-20 11:29:34,213 module.py:788] default: "GET /set?name=a&value=1 HTTP/1.1" 200 5
data: Data(key=Key('Datum', 'fastsimon', 'Data', None), name=u'b', value=u'2')
INFO     2016-10-20 11:29:50,442 module.py:788] default: "GET /set?name=b&value=2 HTTP/1.1" 200 5
before delete
ndb.Key('Datum',q): Key('Datum', 4661104668049408)
ndb.Key('Datum',q): Key('Datum', 5787004574892032)
list_of_keys: [Key('Datum', 4661104668049408), Key('Datum', 5787004574892032)]
after delete
INFO     2016-10-20 11:30:04,125 module.py:788] default: "GET /end HTTP/1.1" 200 4

But, when I go to the Datastore viewer and press the Flush Memcache, I still see entities there:

when I go to the Datastore viewer and press the Flush Memcache, I still see entities there

What should I change so that the NDB entities will be deleted?

2
pythongoogle-app-enginegoogle-cloud-datastoreapp-engine-ndb
you should also print the ancestor key used in your query and check if the entities in the datastore match that ancestor or not - Dan Cornilescu
Your Data.owner_query and EndHandler queries don't match: one is an ancestor query, the other is not, so the results they produce would be non-overlapping. - Dan Cornilescu
The purpose of EndHandler is to delete all entities belonging to me, so - do I still need to check the ancestor, @DanCornilescu? Also, doesn't the fact that nothing is printed following the after delete line shows that the entities were supposedly deleted? - boardrider
And the list_of_keys you build in the EndHandler may contain non-existent keys: when doing ndb.Key('Datum',q.integer_id()) you'll create a key without an ancestor, which won't match the original q key if that had an ancestor. ndb.delete_multi() won't complain if the key doesn't exist. - Dan Cornilescu
Why don't you call directly ndb.delete_multi(qry) instead - since qry already contains the actual keys you want to delete? - Dan Cornilescu

2 Answers

1
votes

Based on @DanCornilescu excellent suggestions, I changed the code for EndHandler, and it deletes entities now like a champ.

Code:

from google.appengine.ext import ndb
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

from models import Data

KEY = "fastsimon"
datum_key = dict()

class InvalidHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Invalid entry')

class GetHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Get!\n')
        name = self.request.get('name')
        out_str = "Should not be seen"
        try:
            ancestor_key = datum_key[name]
            qry = Data.owner_query(ancestor_key)
            data = qry.fetch()
            out_str = data[-1].value
        except KeyError:
            out_str = "None"
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write(out_str)


class EndHandler(webapp.RequestHandler):
    def show_entities(self):
        for q in datum_key.values():
            k = ndb.Key('Datum', 'fastsimon', 'Data',q.integer_id())
            print "entity for",k,"=>", ndb.Key('Datum', 'fastsimon', 'Data',q.integer_id()).get()

    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('End!')
        print "datum_key.values():",datum_key.values()
        print "before delete"
        self.show_entities()
        ndb.delete_multi(datum_key.values())
        print "after delete"
        self.show_entities()


class SetHandler(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Set!\n')
        name=self.request.get('name')
        data = Data(parent=ndb.Key("Datum", KEY),
                    name=name,
                    value=self.request.get('value'))
        ret_key = data.put()
        datum_key[name] = ret_key
        print "data:",data

def main():
    application = webapp.WSGIApplication([
            ('/get', GetHandler),
            ('/set', SetHandler),
            ('/end', EndHandler)],     
            debug=False) 
    global app
    app = application
    run_wsgi_app(app)

if __name__ == 'main':
    main()

debugging lines:

data: Data(key=Key('Datum', 'fastsimon', 'Data', 4943129400573952), name=u'z', value=u'55')
INFO     2016-10-20 13:40:17,008 module.py:788] default: "GET /set?name=z&value=55 HTTP/1.1" 200 5
data: Data(key=Key('Datum', 'fastsimon', 'Data', 6069029307416576), name=u'x', value=u'88')
INFO     2016-10-20 13:40:28,556 module.py:788] default: "GET /set?name=x&value=88 HTTP/1.1" 200 5
datum_key.values(): [Key('Datum', 'fastsimon', 'Data', 4943129400573952), Key('Datum', 'fastsimon', 'Data', 6069029307416576)]
before delete
entity for Key('Datum', 'fastsimon', 'Data', 4943129400573952) => Data(key=Key('Datum', 'fastsimon', 'Data', 4943129400573952), name=u'z', value=u'55')
entity for Key('Datum', 'fastsimon', 'Data', 6069029307416576) => Data(key=Key('Datum', 'fastsimon', 'Data', 6069029307416576), name=u'x', value=u'88')
after delete
entity for Key('Datum', 'fastsimon', 'Data', 4943129400573952) => None
entity for Key('Datum', 'fastsimon', 'Data', 6069029307416576) => None
INFO     2016-10-20 13:40:40,517 module.py:788] default: "GET /end HTTP/1.1" 200 4

and Datastore viewer doesn't show any entities.

1
votes

In general you need to be careful with your queries when you use ancestries for your entities - queries with and without ancestor keys specified will produce different (non-overlapping) results. See Ancestor Queries

The root cause of your problem is that the keys you're building in the list_of_keys to be deleted do not match the original keys if the respective entities have ancestors, thus what's passed to ndb.delete_multi may miss some of the keys you intend to delete.

ndb.Key('Datum',q.integer_id()) produces a key for an entity without an ancestor which will not match the q key if q is the key of an entity with an ancestor.

A simple solution would be to not convert the keys you want to delete, instead pass them directly to ndb.delete_multi(), for example you could simply do ndb.delete_multi([key for key in qry]) instead of ndb.delete_multi(list_of_keys)