How to schedule a Cloud Datastore export with App Engine in Python3

Question

I have an app which stores user data in GCP Datastore. I have made a cron job that is scheduled to export the data in Datastore using the instructions given here.

However, I need to change python2 to python3.

According to docs,app uses a app_identity library to obtain tokens.

    from google.appengine.api import app_identity
    access_token, _ = app_identity.get_access_token('https://www.googleapis.com/auth/datastore')

But this library is not supported in python3 according to here.

How can I get a access_token in python3?

Are you sure that it's not supported ? for me that's just their documentation that is not up-to-date. The app_identity should have its correspondent in python3 — Didi Bear
Importing google.appengine gives error as error in python3:ModuleNotFoundError: No module named 'google.appengine' — yum

Juan Lara Juan Lara · Accepted Answer · 2019-06-01T02:09:53

Checkout the google-api-python-client library. It's supported in python 3 and makes it easy to build requests for the Cloud Datastore API.

Another thing you'll need to change is the webapp2 library since it's also not supported in python 3. You can replace it with something like Flask.

Here's an example of the app rewritten for python 3:

app.yaml

runtime: python37

handlers:
- url: /.*
  script: auto

(deploy to a non-default service if needed with service: service_name)

requirements.txt

Flask
google-api-python-client

main.py

import datetime
import os
from googleapiclient.discovery import build

from flask import Flask, render_template, request

app = Flask(__name__)


@app.route('/')
def hello_world():
    return 'Hello, World!'    

@app.route('/cloud-datastore-export')
def export():
    # Deny if not from the Cron Service
    assert request.headers['X-Appengine-Cron']
    # Deny if output_url_prefix not set correctly
    output_url_prefix = request.args.get('output_url_prefix')
    assert output_url_prefix and output_url_prefix.startswith('gs://')
    timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
    if '/' not in output_url_prefix[5:]:
      # Only a bucket name has been provided - no prefix or trailing slash
      output_url_prefix += '/' + timestamp
    else:
      output_url_prefix += timestamp
    kinds = request.args.getlist('kind')
    namespace_ids = request.args.getlist('namespace_id')
    entity_filter = {
        'kinds': kinds,
        'namespace_ids': namespace_ids 
    }
    body = {
        'output_url_prefix': output_url_prefix,
        'entity_filter': entity_filter
    }
    project_id = os.environ.get('GOOGLE_CLOUD_PROJECT')
    client = build('datastore', 'v1')
    client.projects().export(projectId=project_id, body=body).execute()  
    return 'Operation started' 


if __name__ == '__main__':
    # This is used when running locally only. When deploying to Google App
    # Engine, a webserver process such as Gunicorn will serve the app. This
    # can be configured by adding an `entrypoint` to app.yaml.
    # Flask's development server will automatically serve static files in
    # the "static" directory. See:
    # http://flask.pocoo.org/docs/1.0/quickstart/#static-files. Once deployed,
    # App Engine itself will serve those files as configured in app.yaml.
    app.run(host='127.0.0.1', port=8080, debug=True)

How to schedule a Cloud Datastore export with App Engine in Python3

1 Answers