Querying GCP Datastore backup without overwriting live database

Question

I am trying to access a backup created by gcloud datastore export. I only want to access some entries in the backup and don't want to modify the live database to do so.

So far I have tried:

Restore using gcloud datastore import: it seems like my only option is to limit the namespaces and kinds that are being overwritten from the backup. If I do not want to overwrite anything, I can create a new GCP project and restore to that one. However, this is a tough option because I don't control billing for my organization and the restored backup would exceed the free tier.

Parse the backup files manually: The import export guide mentions:

The output of a managed export uses the LevelDB log format.

I wrote a parser based on the grammar linked above. The good news is that it resembles my data in a hex editor. Unfortunately, it seems like each record is wrapped in some undocumented binary format that also contains several copies of the namespace, kind, and key. I was not sure how to parse this.

Thanks in advance for any help or suggestions!

Kevin Chen Kevin Chen · Accepted Answer · 2021-02-04T23:24:48

Download the backup from the GCS bucket:

mkdir ~/my-datastore-backups
gsutil -m cp -r gs://my-datastore-backups/2020-07-30T09:00:07_35225/ ~/my-datastore-backups

Import the backup to the datastore emulator:

curl -X POST localhost:8081/v1/projects/my-project:import \
-H 'Content-Type: application/json' \
-d '{"input_url":"/home/user/my-datastore-backups/2020-07-30T09:00:07_35225/2020-07-30T09:00:07_35225.overall_export_metadata"}'

Querying GCP Datastore backup without overwriting live database

1 Answers