python - GAE SDK 1.6.4 dev_appserver datastore flush

1
votes

Hoping to get a comment from the GAE python team on this one.

Has something changed between 1.6.3, 1.6.4 with regards to the way the dev_appserver datastore is flushed to disk on app exit?

I'm using django-nonrel, and on 1.6.3, and before, I used to be able to load up a python shell: python manage.py shell (manage.py is provided by django-nonrel)

I could then import my models and acceess the datastore, modify entities, and quit.

On 1.6.4, I'm able to do that, but when I quit, changes are not saved to the datastore. When I run django-nonrel as a WSGI app, it saves properly, and I see a message on exit ("Applying all pending transactions and saving the datastore").

3
pythongoogle-app-enginedjango-nonrel
Something did change. I'll ask around. - Guido van Rossum
Thanks Guido, I hacked in a fix for django-nonrel. I suspect any other python users who want to get at the datastore via the python shell will have a similar issue. - dragonx
See Alfred Fuller's response to stackoverflow.com/questions/10047315/… for an explanation and source links. - Guido van Rossum
@Guido What about handling exit() from the shell? That doesn't send a TERM or KeyInterrupt signal. Is using atexit, like in my answer below correct? - dragonx

3 Answers

3
votes

Thanks to dragonx for his solution and info. I run my devserver from eclipse, and I was amazed to see my data not beeing saved after upgrading to 1.6.4 I added a flush to the database after every web request, to do that I implemented a base class for all requests and override dispatch:

developmentServer = False

if os.environ.get('SERVER_SOFTWARE','').startswith('Development'):
    developmentServer = True

class BaseRequestHandler(webapp2.RequestHandler):
    def dispatch(self):
        retValue = super(BaseRequestHandler, self).dispatch()
        if developmentServer:
                from google.appengine.tools import dev_appserver 
                dev_appserver.TearDownStubs()

        return retValue

informing about a change in behavior like that in the release notes, would have saved me two days of searching what went wrong in my upgrade.

2
votes

It looks like there have been some changes. I've been able to hack around the problem with the following:

from google.appengine.tools import dev_appserver
import atexit
atexit.register(dev_appserver.TearDownStubs)

This ensures the datastore is flushed on exit.

0
votes

Before 1.6.4, we saved the datastore after every write. This method does not work when simulating the transactional model found in the High Replication Datastore (you would lose the last couple writes). It is also horribly inefficient. We changed it so the datastore dev stub flushs all writes and saves it's state on shut down.

Following the code:

  1. https://bitbucket.org/wkornewald/djangoappengine/src/60c2b3339a9f/management/commands/runserver.py#cl-154
  2. http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/dev_appserver_main.py#683

It looks like manage.py should work if the server is shut down cleanly (with a TERM signal or KeyInterrupt).