Using google cloud datastore emulator with dev_appserver

Question

I've been reading between the lines and trying to interface dev_appserver.py with the new 'non-legacy' google cloud datastore emulator.

My main motivation is to integrate my appengine projects with my google cloud dataflow pipeline while I am developing on my local machine.

This is the procedure to setup the integration, as far as I understand:

• Install the googledatastore library with pip (you may need to force an upgrade of six with easy_install particularly if you are using system python El Capitan)

• Using the google cloud sdk tools run the google cloud datastore emulator:

  gcloud beta emulators datastore start --no-legacy
  

• In the terminal where dev_appserver will run the following command to set datastore environment variables:

  $(gcloud beta emulators datastore env-init --no-legacy)
  

• If the project id in app.yaml does not match the currently select project id in the gcloud tools set the following environment variable in the same shell:

  export DATASTORE_USE_PROJECT_ID_AS_APP_ID=true
  

• Run dev_appserver.py and navigate to http://localhost:8000/datastore which should let you navigate the emulator's datastore data.

However this does not work so smoothly when I navigate to the url I get:

BadArgumentError: Could not import googledatastore.
This library must be installed with version >= 4.0.0.b1 to use the Cloud Datastore 
API.

This is strange because if I open a python shell and run import googledatastore no error occurs.

If I dig a bit deeper and instrument the import code in dev_appserver and log the error here I get the following traceback:

Traceback (most recent call last):
  File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/datastore/datastore_pbs.py", line 52, in <module>
    import googledatastore
  File "/Library/Python/2.7/site-packages/googledatastore/__init__.py", line 21, in <module>
    from . import helper
  File "/Library/Python/2.7/site-packages/googledatastore/helper.py", line 25, in <module>
    from google.datastore.v1beta3 import entity_pb2
ImportError: No module named datastore.v1beta3 

I also have no issue importing google.datastore.v1beta3 in a regular python shell.

Even stranger if I run PYTHONINSPECT=x dev_appserver.py app.yaml and drop out into the shell executing these imports runs without error. Perhaps there is something odd going on with the python path while dev_appserver.py is starting?

Can anybody tell me how to get this feature working?

UPDATE: I reproduced this problem on ubuntu 14.04 (system python 2.7.6, pip 8.1.2 via easy_install, gcloud-sdk 118.0.0, app-engine-python 1.9.38) as well as OS X (gcloud sdk 114.0.0, app-engine-python 1.9.38, system python 2.7.10).

Answer 1

Heads up folks,

Using google cloud datastore emulator with dev_appserver is now available! (link)

Update Google Cloud SDK, then run dev_appserver with '----support_datastore_emulator'.

This feature is in Beta, welcome to try it! We are actively collecting feedback.

Answer 2

Actually gcloud datastore emulator and dev_appserver points to two different endpoints. the localhost:8000 is the default dev_appserver admin console, while the datastore emulator have a different console url which could be found in print outs when it starts.

I assume the admin console you are accessing belongs to dev_appserver, then the issue should be within dev_appserver. Could you attach the code snippet(if there is any) using datastore api in dev_appserver? BTW it should be gcloud.datastore instead of appengine.ext.(n)db talking to gcloud datastore-emulator.

Also, I'm curious what would happen if you do not start datastore-emulator with '--no-legacy', or even do not start datastore-emulator but just start dev_appserver?

Answer 3

We recently ran into this same issue. One thing to look at is the output of the command:

(gcloud beta emulators datastore env-init --no-legacy)

The problem we had was that when we ran the emulator the emulator was choosing say port 8607 but the env-init method was returning a different port 8328.

So, what I would recommend is to start the emulator and see what port it is running on:

[datastore] Aug 04, 2016 3:50:50 PM com.google.appengine.tools.development.AbstractModule startup
[datastore] INFO: Module instance default is running at http://localhost:8607/
[datastore] Aug 04, 2016 3:50:50 PM com.google.appengine.tools.development.AbstractModule startup
[datastore] INFO: The admin console is running at http://localhost:8607/_ah/admin

In this case 8607 and then fire off the env-init method to get the syntax but validate the port. In our case with the above server running the env-init returns 8328

$ (gcloud beta emulators datastore env-init)
export DATASTORE_DATASET=my-app
export DATASTORE_EMULATOR_HOST_PATH=localhost:8328/datastore
export DATASTORE_EMULATOR_HOST=localhost:8328
export DATASTORE_HOST=http://localhost:8328
export DATASTORE_PROJECT_ID=my-app

So change this to the correct port:

export DATASTORE_DATASET=my-app
export DATASTORE_EMULATOR_HOST_PATH=localhost:8607/datastore
export DATASTORE_EMULATOR_HOST=localhost:8607
export DATASTORE_HOST=http://localhost:8607
export DATASTORE_PROJECT_ID=my-app

Then use this where your project is running and you should be good to go. This is what fixed it for us. Hope that helps!