Faster App Engine Development Datastore Alternative

Question

Is there a way to use a real database(SQLite, Mysql, or even some non-relational one) as datastore for development, instead of memory/file datastore that is provided.

I saw few projects, GAE-SQLite(did not seem to be working) and one tip about accessing production datastore using remote api (still pretty slow for large datasets).

Easier development, development datastore shipped with SDK is too slow to experiment even with more than few hundreds of entities. — M. Utku ALTINKAYA
Apart from my answer below, I did not experience the significant slowdowns you mention with lots of entities. — Koen Bok
With about 40k entities my development server consumes about 2GB of RAM bringing my entire computer to a halt. Is there anyway we can use the real online datastore with the development server? — speedplane
With a few thousand entities, each with a single property, my dev server takes so long to call Entity.query that I just leave it and get on with something else for a while. I've taken to running a clean up in every test, just to prevent anything accumulating, so now every test takes ages :( — Carl Smith

Koen Bok Koen Bok · Accepted Answer · 2010-01-22T09:54:21

MongoDB works great for that. You will need:

The MongoDB stub: http://github.com/mongodb/mongo-appengine-connector
MongoDB: http://www.mongodb.org/display/DOCS/Downloads
Some code to set it up like:

code:

import datastore_mongo_stub

os.environ['APPLICATION_ID'] = 'test'

datastore = datastore_mongo_stub.DatastoreMongoStub(
    os.environ['APPLICATION_ID'], 'woot', '', require_indexes=False)

apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', datastore)

But if you're looking for truly faster development (like I was) the datastore is actually not the issue as much is the single threaded web server. I tried to replace it with spawning but that was a little too hard. You could also try to set up TyphoonAE which will mimic the appengine stack with open alternatives.

Be aware that if you do any of these, you might lose some of the exact behavior the current tools provide, meaning that if you deploy you could get results you didn't expect. In other words; make sure you know what you're doing :-)

Faster App Engine Development Datastore Alternative

3 Answers