Is it possible to start two dev_appserver.py connecting to the same google cloud datastore emulator?

2
votes

Use case: I am developing an appengine standard application in python, and another one in go. I'd like to have the entities from both the applications in the same datastore. Is that possible?

When I start the first dev_appserver.py with the flag --support_datastore_emulator=true, and with a specific --datastore_path, the gcloud beta emulators datastore env-init command fails with a ERROR: (gcloud.beta.emulators.datastore.env-init) Unable to find env.yaml in the data_dir [~/.config/gcloud/emulators/datastore]. Please ensure you have started the appropriate emulator.

1
google-app-enginegoogle-cloud-datastore
Over the last ffew days, I have had a look at the dev_appserver.py. I have discovered that there are both documented and undocumented options concerning the datastore emulator. At last, I have been able to execute two dev_appservers with the same emulator (or at least I think so, because the datastore file got corrupted at the end of the test :-)). My solution was to pass the --datastore_emulator_port COMMON_PORT option to both dev_appserver instances. – nilleb
In the datastore indexes section of the two applications amin interfaces (usually found at localhost:8000) I was seeing the indexes of both applications, but the datastore viewer was showing only the entities concerning the current application. And, when I tried to stop and restart the servers, the datastore was not anymore usable. Thus, the test wasn't successful in my opinion. – nilleb
You comment that you have managed to start the two dev_appserver.py sesions. Could you please post what you commented about how you managed to do it as an answer? Thank you. – Rodrigo C.

1 Answers

4
votes

Yes, it is possible, but you need to carefully set it up.

First thing to keep in mind is that only one emulation process (be it the emulator itself or the development server) should handle a certain datastore emulation dir, running multiple of them simultaneously will most likely cause corruption of the data, as you may have observed trying to run 2 development servers sharing the same --datastore_path config.

So you want just one datastore emulator process handling the storage dir. You do not want the development server(s) - which are also capable or running their own datastore emulation, but in a standalone manner - to also handle that storage dir, so you shouldn't be using the --datastore_path option for them. The development server(s) should instead just talk to the datastore emulator process for all their datastore accesses.

When you start the datastore emulator process you'll see in its log a line specifying the DATASTORE_EMULATOR_HOST environment variable that you need to pass to all the development servers that you want to share that datastore emulation instance:

[datastore] API endpoint: http://0.0.0.0:5555
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore] 
[datastore]   export DATASTORE_EMULATOR_HOST=0.0.0.0:5555
[datastore] 
[datastore] Dev App Server is now running.

So set that environment variable in the shell(s) where you want to start your development server(s), then start the servers with just these datastore related options:

export DATASTORE_EMULATOR_HOST=0.0.0.0:555
[.../]dev_appserver.py --support_datastore_emulator=true ...

This env var is the only thing needed you don't need to run gcloud beta emulators datastore env-init here, In the server's log you should see a line like this:

WARNING 2018-06-14 13:54:41,238 api_server.py:581] Detected environment variable DATASTORE_EMULATOR_HOST=0.0.0.0:5555, dev_appserver will speak to the Cloud Datastore emulator running on this address. The datastore_path /some_path_you_may_have_used_before/datastore.db will be neglected. If you want datastore to store on /some_path_you_may_have_used_before/datastore.db, remove DATASTORE_EMULATOR_HOST from environment variables and restart dev_appserver

When a development server makes its first datastore access you'll see a few entries in the datastore emulator log, like these:

[datastore] Jun 14, 2018 10:02:53 AM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[datastore] INFO: Adding handler(s) to newly registered Channel.
[datastore] Jun 14, 2018 10:02:53 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[datastore] INFO: Detected HTTP/2 connection.

If the DATASTORE_EMULATOR_HOST doesn't point to a running datastore emulator process (or there's something wrong with the communication with it) you'll see errors in the development server's logs, probably similar with these:

ERROR    2018-06-14 14:39:08,026 api_server.py:373] Exception while handling datastore_v3.Get()
Traceback (most recent call last):
  File "/home/usr_local/google-cloud-sdk-204.0.0/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 333, in _handle_POST
    response = service_stub.MakeSyncCallForRemoteApi(request)
  File "/home/usr_local/google-cloud-sdk-204.0.0/platform/google_appengine/google/appengine/tools/devappserver2/datastore_grpc_stub.py", line 190, in MakeSyncCallForRemoteApi
    request_pb, _TIMEOUT)
  File "/home/usr_local/google-cloud-sdk-204.0.0/platform/google_appengine/lib/grpcio-1.9.1/grpc/_channel.py", line 487, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/home/usr_local/google-cloud-sdk-204.0.0/platform/google_appengine/lib/grpcio-1.9.1/grpc/_channel.py", line 437, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
_Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>

More info at:

NOTE: that (at least presently) only the Google Cloud SDK development server (a recent version, I tested with 204.0.0) supports talking to a standalone datastore emulator, the GAE-specific SDK ones don't (or at least the 1.9.69 python one I'm presently using doesn't). From Migrating to the Cloud Datastore Emulator:

Note: This migration requires the use of the Google Cloud SDK-based tooling.