Google Datastore Emulator using Java (Not using GAE)

Question

I am using Google Cloud's Datastore Client Library for Java to access the Cloud Datastore.

Note: I am not using App Engine to deploy my application; just running a local application for development purposes.

Following the example, I can read/write to the Cloud Datastore.

Datastore datastore = DatastoreOptions.defaultInstance().service();
KeyFactory keyFactory = datastore.newKeyFactory().setKind("MyKind");
Key key = keyFactory.newKey();
Entity entity = datastore.get(key);

I want to be able to write to a local Datastore emulator instance instead. Following the guide here, I run gcloud beta emulators datastore start. This shows up in my terminal:

C:\Users\User>gcloud beta emulators datastore start
WARNING: Reusing existing data in [C:\Users\User\AppData\Roaming\gcloud\emulators\datastore].
Executing: cmd /c C:\Users\User\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\cloud-datastore-emulator\cloud_datastore_emulator.cmd start --host=localhost --port=8964 --store_on_disk=True --consistency=0.9 --allow_remote_shutdown C:\Users\User\AppData\Roaming\gcloud\emulators\datastore
[datastore] Oct 31, 2016 11:37:27 AM com.google.cloud.datastore.emulator.CloudDatastore$FakeDatastoreAction$7 apply
[datastore] INFO: Provided --allow_remote_shutdown to start command which is no longer necessary.
[datastore] Oct 31, 2016 11:37:27 AM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub <init>
[datastore] INFO: Local Datastore initialized:
[datastore]     Type: High Replication
[datastore]     Storage: C:\Users\User\AppData\Roaming\gcloud\emulators\datastore\WEB-INF\appengine-generated\local_db.bin
[datastore] Oct 31, 2016 11:37:28 AM io.grpc.internal.ManagedChannelImpl <init>
[datastore] INFO: [ManagedChannelImpl@5e955596] Created with target localhost:8964
[datastore] Oct 31, 2016 11:37:28 AM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub load
[datastore] INFO: The backing store, C:\Users\User\AppData\Roaming\gcloud\emulators\datastore\WEB-INF\appengine-generated\local_db.bin, does not exist. It will be created.
[datastore] Oct 31, 2016 11:37:28 AM io.gapi.emulators.netty.NettyUtil applyJava7LongHostnameWorkaround
[datastore] INFO: Unable to apply Java 7 long hostname workaround.
[datastore] API endpoint: http://localhost:8964
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore]
[datastore]   export DATASTORE_EMULATOR_HOST=localhost:8964
[datastore]
[datastore] Dev App Server is now running.
[datastore]

I open another terminal and set the environment variables:

C:\Users\User>gcloud beta emulators datastore env-init > set_vars.cmd && set_vars.cmd
C:\Users\User>set DATASTORE_DATASET=my-project-id
C:\Users\User>set DATASTORE_EMULATOR_HOST=localhost:8964
C:\Users\User>set DATASTORE_EMULATOR_HOST_PATH=localhost:8964/datastore
C:\Users\User>set DATASTORE_HOST=http://localhost:8964
C:\Users\User>set DATASTORE_PROJECT_ID=my-project-id

I run my application and make a REST call to post or retrieve an Entity, but this only reads/writes against the Cloud Datastore. Heading to localhost:8964/datastore gives me Not Found. Although starting the emulator tells me that it created local_db.bin file, the folder supposedly containing it is empty. I also want to stay away from using LocalDatastoreHelper in order to access the local emulator. Is there any way to achieve using gcloud only ?

Answer 1

The line below always connects to the remote datastore. Uses the default options (e.g. project, auth credentials) from gcloud settings.

Datastore datastore = DatastoreOptions.defaultInstance().service();

To connect to the local datastore, try the below:

@Test
public void test1() throws IOException, InterruptedException {
    Datastore ds = DatastoreOptions.builder().host("http://localhost:9999").projectId("my-project").build().service();
    com.google.cloud.datastore.Key key = ds.newKeyFactory().kind("MyEntity").newKey("mykey");
    com.google.cloud.datastore.Entity entity = com.google.cloud.datastore.Entity.builder(key).set("p1", "Hello World!").build();
    entity = ds.put(entity);
    entity = ds.get(key);
    System.out.println(entity);
}

I started my Datastore Emulator on localhost:9999. Set that as the host when building the DatastoreOptions.

I've confirmed that the Emulator console shows requests are received and entities are persisted. I've also checked the data file (local_db.bin) and it shows the data (of course it is not a plain text file).

The one thing I don't know is - if there is a way to manage the local datastore using a browser interface. I could not find much documentation on how to administer the local datastore just like how we do the remote one from Cloud Console. Perhaps someone else can help with this.

Answer 2

I assume that you want to test against the Datastore Emulator. In that case it is not necessary to start the Datastore Emulator from the shell. There is a LocalDatastoreHelper in the gcloud library that allows you to create, start, reset and stop a local Datastore Emulator with ease.

I didn't found any documentation on that, so I created this testcases for you:

import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;
import com.google.cloud.datastore.Entity;
import com.google.cloud.datastore.KeyFactory;
import com.google.cloud.datastore.testing.LocalDatastoreHelper;
import org.junit.*;

import java.io.IOException;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;

/**
 * This testcase demonstrate the use of the datastore emulator in JUnit test cases.
 *
 * from @link https://www.kontaktlinsen-preisvergleich.de
 */
public class DatastoreEmulatorTest {

    protected static LocalDatastoreHelper localDatastoreHelper;

    protected Datastore datastore;

    protected KeyFactory keyFactory;

    @BeforeClass
    public static void setUpClass() throws InterruptedException, IOException {
        // create and start a local datastore emulator on a random free port
        // this also means that you probably can run tests like this concurrently.
        System.out.println("[Datastore-Emulator] start");
        localDatastoreHelper = LocalDatastoreHelper.create();
        localDatastoreHelper.start();
        System.out.println("[Datastore-Emulator] listening on port: " + localDatastoreHelper.getPort());

        // set the system property to tell the gcloud lib to use the datastore emulator
        System.setProperty("DATASTORE_EMULATOR_HOST","localhost:" + localDatastoreHelper.getPort());
    }

    @Before
    public void setUp() {
        // create the datastore instance
        // because of the system property set it in setUpClass() this
        // datastore will be connected with the datastore emulator.
        datastore = DatastoreOptions.getDefaultInstance().getService();
        keyFactory = datastore.newKeyFactory().setKind("TestEntity");
    }

    @After
    public void tearDown() throws IOException {
        System.out.println("[Datastore-Emulator] reset");
        // this resets the datastore after every test
        localDatastoreHelper.reset();
    }

    @AfterClass
    public static void tearDownClass() throws InterruptedException, IOException {
        System.out.println("[Datastore-Emulator] stop");
        // this stops the datastore emulator after all tests are done
        localDatastoreHelper.stop();
    }

    @Test
    public void test1() {
        // stores an entity in the datastore and retrieves it later

        // create an Entity "TestEntity"
        Entity.Builder builder = Entity.newBuilder(keyFactory.newKey(42));
        builder.set("name", "Test1");

        // store it in datastore
        datastore.put(builder.build());

        // retrieve entity by key
        Entity entity = datastore.get(keyFactory.newKey(42));
        assertNotNull(entity);
        assertEquals("Test1", entity.getString("name"));
    }

    @Test
    public void test2() {
        // try to access the entity created in test1, shouldn't work because
        // of calling reset in tearDown() after each test.

        // try to retrieve entity by key
        Entity entity = datastore.get(keyFactory.newKey(42));
        assertNull(entity);
    }
}

The LocalDatastoreHelper creates a Datastore Emulator instance on a free port and doesn't store to disk - when you stop the testcase with the debugger and look for processes, you'll find something like this:

$ ps ax | grep CloudDatastore
2614   ??  R      0:01.39 /usr/bin/java -cp /Users/marco/google-cloud-sdk/platform/cloud-datastore-emulator/CloudDatastore.jar com.google.cloud.datastore.emulator.CloudDatastore /Users/marco/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=localhost --port=57640 --store_on_disk=False --consistency=0.9 --allow_remote_shutdown /var/folders/ky/c126qk_161159ltyrbpdxv8w0000gn/T/gcd2141205756617995044

That also means you should also be able to run tests in parallel, too.