OS X AppEngine - ImportError: No module named _sqlite3

Question

UPDATE: This is a problem I am having with the 1.8.0 App Engine SDK on a fresh install of OS X 10.8.3.

First up - there's a bunch of questions on SO with a similar title. I've checked them out, and I don't believe they answer my question. Mostly they recommend getting libsqlite3-dev and rebuilding python to get _sqlite3.so, but that's already where it should be:

$ find / -name _sqlite3.so
Password:
...
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload/_sqlite3.so
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/_sqlite3.so
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_sqlite3.so

The actual code that that causes app engine SDK to try to load that module is:

remaining = TaskSetElement.all().filter('taskSet', ts_key).filter('complete', False).count()

Here's the SDK stack trace:

  File "~/dev/myApp/myApp/task.py", line 90, in completeTaskSetElement
    remaining = TaskSetElement.all().filter('taskSet', ts_key).filter('complete', False).count()
  File "~/dev/GAE/google_appengine/google/appengine/ext/db/__init__.py", line 2133, in count
    result = raw_query.Count(limit=limit, **kwargs)
  File "~/dev/GAE/google_appengine/google/appengine/api/datastore.py", line 1698, in Count
    batch = self.GetBatcher(config=config).next()
  File "~/dev/GAE/google_appengine/google/appengine/datastore/datastore_query.py", line 2754, in next
    return self.next_batch(self.AT_LEAST_ONE)
  File "~/dev/GAE/google_appengine/google/appengine/datastore/datastore_query.py", line 2791, in next_batch
    batch = self.__next_batch.get_result()
  File "~/dev/GAE/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
    return self.__get_result_hook(self)
  File "/Users/colin/dev/GAE/google_appengine/google/appengine/datastore/datastore_query.py", line 2528, in __query_result_hook
    self._batch_shared.conn.check_rpc_success(rpc)
  File "~/dev/GAE/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1222, in check_rpc_success
    rpc.check_success()
  File "~/dev/GAE/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 570, in check_success
    self.__rpc.CheckSuccess()
  File "/Users/colin/dev/GAE/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl
    self.request, self.response)
  File "~/dev/GAE/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall
    self._MakeRealSyncCall(service, call, request, response)
  File "~/dev/GAE/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 234, in _MakeRealSyncCall
    raise pickle.loads(response_pb.exception())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/__init__.py", line 24, in 
    from dbapi2 import *
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/dbapi2.py", line 27, in 
    from _sqlite3 import *
  File "~/dev/GAE/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 856, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named _sqlite3

I've got a bunch of datastore code prior to this line that's executing fine. I get the same problem running dev_appserver.py directly from the command line or in eclipse with pydev.

From the command line, everything looks good:

$ python
Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> import _sqlite3
>>> import sys
>>> print(sys.path)
['', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages']
>>>

This code snippet (running in app engine SDK) removes the app engine datastore code from the equation:

...
logging.info("Python Version: %s" % sys.version)
logging.info(filter(lambda p: 'lib-dynload' in p, sys.path))

import sqlite3
...

It outputs this:

INFO     2013-05-26 05:55:12,055 main.py:38] Python Version: 2.7.2 (default, Oct 11 2012, 20:14:37) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)]
INFO     2013-05-26 05:55:12,055 main.py:40] ['/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']
ERROR    2013-05-26 05:55:12,058 cgi.py:121] Traceback (most recent call last):
  File "main.py", line 42, in 
    import sqlite3
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/__init__.py", line 24, in 
    from dbapi2 import *
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sqlite3/dbapi2.py", line 27, in 
    from _sqlite3 import *
  File "~/dev/GAE/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 856, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named _sqlite3

Any ideas what the problem is? Thanks,

Colin

Answer 1

It looks like adding '_sqlite3' to the _WHITE_LIST_C_MODULES list at line 742 in sandbox.py (which lives at /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python on my machine) has worked for me for now.

To my uneducated eye it looks like CModuleImportHook is incorrectly blocking the import of _sqlite3 and that modifying any one of the checks it has to return None removes the error. Someone with more nous - please expand on what I've said or correct me!

That change may allow you to import _sqlite3 in your own code, which would be a mistake. Perhaps it needs a way to limit the import to dbapi2.py?

Answer 2

From you stack trace it shows you are trying to import sqlite in your main.py.

Why are you doing this?

Importing sqlite is not supported in appengine. sqlite is implemented with a binary library and you can't just import any old binary. In addition the filesystem is read only, you couldn't write to sqlite db.

Answer 3

You mention this is a fresh install of OSX. I wonder if you used Homebrew to build Python? Doing Brew-update or brew-upgrade caused me similar problems to this, although not on GAE. You might find https://github.com/mxcl/homebrew/issues/17312 useful.