How to connect Google Cloud Library for Datastore to the App Engine development server?

Question

We are planning the migration of an internal app running on Google App Engine Standard Environment for Java 8 from the now superseded App Engine APIs to the recommended client library for Cloud Firestore in Datastore Mode, also in order to enable porting to other execution environments.

Besides a host of issues with missing IN/OR query operators, we are also struggling with the setup for local testing: according to Using the Java 8 Local Development Server

The development web server simulates Datastore using a local file-backed Datastore on your computer. The Datastore is named local_db.bin, and it is created in your application's WAR directory, in the WEB-INF /appengine-generated/ directory.

but we have no clue about how to connect Google Cloud Client Library for Datastore to the local emulator.

Defining default credentials with

gcloud auth application-default login

or setting the GOOGLE_APPLICATION_CREDENTIALS environment variable after obtaining the credentials for the service account with something like

gcloud iam service-accounts keys create key.json \
    [email protected]

as suggested elsewhere, just results in the client library connecting to the actual cloud server, rather than to the local emulator, as per the ADC policy.

I'd expect the development server to automatically provide connection hints to the client library, but that's apparently not the case.

Any suggestion for setting up a local testing environment, taking into account that we can't just migrate to the standalone Datastore Emulator, as we need other services currently provided only by the App Engine development server (e.g. email submission)?

Edit / After further tinkering we are working around the issue by using both the Local Development Server and the standalone Datastore Emulator as:

gcloud beta emulators datastore start \
  —project=project-id \
  --host-port=localhost:8081 \
  --data-dir=target/war

DATASTORE_EMULATOR_HOST=localhost:8081 java_dev_appserver.sh \
 --port=8080 \
 target/war

However, the process is quite cumbersome and difficult to automate: what we are looking for is a way to automatically connect Google Client Library to the Datastore Emulator managed by the Java 8 Local Development Server when launching the app with something like the App Engine Maven plugin, e.g. mvn appengine:run.

pessolato pessolato · Accepted Answer · 2019-10-02T15:51:17

This GitHub issue was closed with the confirmation that the Datastore Client Library is not compatible with the local Web Server Datastore Emulator.

I actually tried it, to see if was possible to force a connection to the local Web Server. The code below sets a custom builder with the desired host configuration:

DatastoreOptions.Builder builder = DatastoreOptions.newBuilder();
builder.setHost("http://localhost:8080");
builder.setProjectId("<PROJECT_ID>");

Datastore ds = builder.build().getService();

Key key = ds.newKeyFactory().setKind("MyEntity").newKey("mykey");
Entity entity = Entity.newBuilder(key).set("p1", "Hello World!").build();

entity = ds.put(entity);
System.out.println(entity);

entity = ds.get(key);
System.out.println(entity);

After running the local Web Server I noticed a connection was indeed possible, however the Datastore Client Library returned the following error when trying to store new entities:

[INFO] GCLOUD: com.google.cloud.datastore.DatastoreException: Non-protobuf error: <html><head><title>Error 404</title></head>|<body><h2>Error 404</h2></body>|</html>. HTTP status code was 404.

With the following output by the Web Server:

Oct 02, 2019 3:05:59 PM com.google.appengine.tools.development.jetty9.LocalResourceFileServlet doGet
WARNING: No file found for: /v1/projects/<PROJECT_ID>:commit

I believe this adds to the confirmation that the new library is just not compatible to the old emulator.

The workaround you found is probably the best solution while you work/wait on the full migration to the Datastore mode Emulator.

How to connect Google Cloud Library for Datastore to the App Engine development server?

1 Answers