appcfg.py upload_data fails in Google App Engine SDK 1.8.2

Question

I'm working with Google App Engine SDK version 1.8.2 on a Mac OS X 10.6.8 system. The application itself is written in Python 2.7, although I don't think that's relevant to the problem. When I attempt to bulkload the datastore in the development environment, it fails. The command I'm using is:

appcfg.py upload_data --config_file=bulkloader.yaml --kind=Capitulary --url=http://localhost:8080/_ah/remote_api --filename=upload.xml

This command worked with every version of the SDK going back to 2010. It broke when SDK version 1.8.0 came out, it worked again when 1.8.1 came out, and broke again when 1.8.2 came out.

Thanks in advance for your help.

Error message below:

INFO 2013-08-05 14:19:05,312 module.py:595] default: "GET /_ah/remote_api?rtok=215098893971 HTTP/1.1" 200 70 INFO 2013-08-05 14:19:05,569 module.py:595] default: "GET /_ah/remote_api HTTP/1.1" 200 42 ERROR 2013-08-05 14:19:05,716 handler.py:384] Exception while handling service_name: "datastore_v4" method: "AllocateIds" request: "" Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/handler.py", line 380, in post response_data = self.ExecuteRequest(request) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/handler.py", line 411, in ExecuteRequest response_data) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall return stubmap.MakeSyncCall(service, call, request, response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 328, in MakeSyncCall rpc.CheckSuccess() File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl self.request, self.response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall self._MakeRealSyncCall(service, call, request, response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 234, in _MakeRealSyncCall raise pickle.loads(response_pb.exception()) AttributeError: 'NoneType' object has no attribute 'THREADSAFE' INFO 2013-08-05 14:19:05,773 module.py:595] default: "POST /_ah/remote_api HTTP/1.1" 200 96 ERROR 2013-08-05 14:19:05,793 handler.py:384] Exception while handling service_name: "datastore_v4" method: "AllocateIds" request: "" Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/handler.py", line 380, in post response_data = self.ExecuteRequest(request) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/handler.py", line 411, in ExecuteRequest response_data) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall return stubmap.MakeSyncCall(service, call, request, response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 328, in MakeSyncCall rpc.CheckSuccess() File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl self.request, self.response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall self._MakeRealSyncCall(service, call, request, response) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 234, in _MakeRealSyncCall raise pickle.loads(response_pb.exception()) AttributeError: 'NoneType' object has no attribute 'THREADSAFE' INFO 2013-08-05 14:19:05,802 module.py:595] default: "POST /_ah/remote_api HTTP/1.1" 200 96

Answer 1

TL; DR: I 'hacked' around this error by adding return response.ParseFromString(response_pb.response()) in google\appengine\ext\remote_api\remote_api_stub.py on line 228

I have similar error in SDK 1.8.3, here's the Traceback:

Traceback (most recent call last):

  File "C:\Dev\GAE\google\appengine\ext\remote_api\handler.py", line 380, in post
    response_data = self.ExecuteRequest(request)

  File "C:\Dev\GAE\google\appengine\ext\remote_api\handler.py", line 411, in ExecuteRequest
    response_data)

  File "C:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 94, in MakeSyncCall
    return stubmap.MakeSyncCall(service, call, request, response)

  File "C:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 328, in MakeSyncCall
    rpc.CheckSuccess()

  File "C:\Dev\GAE\google\appengine\api\apiproxy_rpc.py", line 156, in _WaitImpl
    self.request, self.response)

  File "C:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall
    self._MakeRealSyncCall(service, call, request, response)

  File "C:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 234, in _MakeRealSyncCall
    raise pickle.loads(response_pb.exception())

AttributeError: 'NoneType' object has no attribute 'THREADSAFE'

I've kinda 'hacked' around it (: In remote_api_stub.py on line 228 I've added return response.ParseFromString(response_pb.response()), thus ignoring problematic error check. upload_data works now, but let's hope they fix this in next release (;

return response.ParseFromString(response_pb.response())
if response_pb.has_application_error():
  error_pb = response_pb.application_error()
  raise apiproxy_errors.ApplicationError(error_pb.code(),
                                         error_pb.detail())
elif response_pb.has_exception():
  raise pickle.loads(response_pb.exception())
elif response_pb.has_java_exception():
  raise UnknownJavaServerError("An unknown error has occured in the "
                               "Java remote_api handler for this call.")
else:
  response.ParseFromString(response_pb.response())

Answer 2

We are aware of this issue an working on a fix. We're tracking this at appengine issue 9666, please watch that thread for updates. Glad you've got a workaround for now. This won't affect bulkload ops against production.

Answer 3

I ran into this same problem with SDK 1.8.5 and can verify that it now works with 1.8.6!