Run appengine backup from task queue

2

votes

In the 1.8.4 release of Google App Engine it states:

A Datastore Admin fix in this release improves security by ensuring that scheduled backups can now only be started by a cron or task queue task. Administrators can still start a backup by going to the Datastore Admin in the Admin Console.

The way to run scheduled backups with cron is documented, but how can we initiate backups from a task queue task?

Are there any other ways to programmatically run backup tasks?

google-app-enginecrondatabase-backups

7

votes

You can create a task queue task with method GET and URL "/_ah/datastore_admin/backup.create" with your parameters specified as arguments to the URL and target the task to run on the 'ah-builtin-python-bundle' version. Example:

url = '/_ah/datastore_admin/backup.create?filesystem=blobstore&name=MyBackup&kind=Kind1&kind=Kind2'
taskqueue.add(
        url=url,
        target='ah-builtin-python-bundle',
        method='GET',
        )

I have cron jobs that trigger my own handlers that then look up a config and create a task queue backup based on that config. This lets me change backup settings without having to update cron jobs and lets me have a lot more control.

The options you can specify in the URL are the same as the documentation describes for CRON job backups so you can specify namespace and gs-bucket-name as well.

I believe to do this in Java you have to create a queue with a target in the queue definition and add your tasks to that queue.

2

votes

I did this by combining Bryce' solution with the code from googles scheduled backup documentation. This way, I'm still using cron.yaml but I have the flexibility for differences in each environment (e.g. don't run a backup in the dev/stage branch based on config in the datastore, don't specify types in the URL that haven't made it out of dev yet).

I also was able to generate the &kind=xxx pairs using this:

    from google.appengine.ext.ndb import metadata
    backup_types = "".join(["&kind=" + kind for kind in metadata.get_kinds()  if not kind.startswith("_")])

The steps were pretty simple in retrospect.

Setup:

a) Enable your default cloud storage bucket

b) Enable datastore admin

Steps:

Add a cron job to kick off the backup (cron.yaml):

cron:
- description: daily backup
  url: /taskqueue-ds-backup/
  schedule: every 24 hours from 21:00 to 21:59

Add a queue to process the tasks (queue.yaml):

- name: ds-backup-queue
  rate: 1/s
  retry_parameters:
    task_retry_limit: 1

Add a route to the task queue handler:

routes = [...,
          RedirectRoute('/taskqueue-ds-backup/',
             tasks.BackupDataTaskHandler, 
             name='ds-backup-queue', strict_slash=True), ...]

Add the handler to process the enqueued items:

from google.appengine.api import app_identity
from google.appengine.api import taskqueue
from google.appengine.ext.ndb import metadata
import config

class BackupDataTaskHandler(webapp2.RequestHandler):
    def get(self):
        enable_ds_backup = bool(config.get_config_setting("enable_datastore_backup", default_value="False"))
        if not enable_ds_backup:
            logging.debug("skipping backup. backup is not enabled in config")
            return

        backup_types = "".join(["&kind=" + kind for kind in metadata.get_kinds() if not kind.startswith("_")])

        file_name_prefix = app_identity.get_application_id().replace(" ", "_") + "_"
        bucket_name = app_identity.get_default_gcs_bucket_name()

        backup_url = "/_ah/datastore_admin/backup.create?name={0}&filesystem=gs&gs_bucket_name={1}{2}".format(file_name_prefix, bucket_name, backup_types)
        logging.debug("backup_url: " + backup_url)

        # run the backup as a service in production. note this is only possible if you're signed up for the managed backups beta.
        if app_identity.get_application_id() == "production-app-id":
            backup_url += "&run_as_a_service=T"

        taskqueue.add(
                url=backup_url,
                target='ah-builtin-python-bundle',
                method='GET',
                )
        logging.debug("BackupDataTaskHandler completed.")

Run appengine backup from task queue

2 Answers