Python: Save data in google cloud datastore emulator

9
votes

I am trying to test google's cloud datastore locally. My test steps are:

  1. Crawl data using Scrapy
  2. Save data to the cloud datastore using local emulator

I followed all the steps in order to use the local emulator

  1. start local emulator: gcloud beta emulators datastore start
  2. set local environment variables: gcloud beta emulators datastore env-init

However, in python, when use the following command to access cloud datastore, it always save the data directly to google cloud instead of saving them to the local emulators

#Imports the Google Cloud client library
from google.cloud import datastore

# Instantiates a client
datastore_client = datastore.Client()

sample_entry = some_data

# Saves the entity
datastore_client.put(sample_entry)

It seems like you cannot specify the library to use the local datastore emulator, just like what they offer in their Node.js client

var datastore = gcloud.datastore({
        apiEndpoint: "http://localhost:8380"
});

My question is, How can I ask the google cloud datastore python library to use local emulator instead of using the cloud directly

2
pythongoogle-app-enginegoogle-cloud-datastoredatastore
What's the value of the DATASTORE_EMULATOR_HOST environment variable after you call env-init? – Ed Davisson
While I'm always one for local testing and dev serving, the Google Cloud environment is so hopelessly convoluted that I definitely wouldn't recommend using their platform emulation packages. Bugs like this, as well as inconsistencies in behavior with the actual deployment environment, make it a headache not worth having; unless there's a reason you can't, I'd suggest just using a staging project in the cloud environment. – kungphu
Probably, you may want to double check your emulator setup following steps given in here: cloud.google.com/datastore/docs/tools/datastore-emulator – Jadav Bheda
Not sure if this is going to help you either: github.com/GoogleCloudPlatform/google-cloud-python/issues/1837 – Jadav Bheda

2 Answers

1
votes

You need to eval $(gcloud beta emulators datastore env-init).

gcloud beta emulators datastore env-init only prints the commands that set the necessary environment variables.

0
votes

You can try something like 

if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
  # Production
else:
  # Local development server

You can follow more here, https://cloud.google.com/appengine/docs/standard/python/tools/using-local-server