Copy Production Cloud Datastore to Dev Server Datastore

5
votes

According to the online documentation, in order to backup data you must go to the Datastore Admin after you have enabled it. It is enabled on my production server.

The to import you simply go to the admin page again, and select the back name you just created. I have downloaded the backup from my Google Bucket onto a local drive, following all the steps of this handy guide found here http://gbayer.com/big-data/app-engine-datastore-how-to-efficiently-export-your-data/

Now that I have the data locally, I am stuck in trying to get it into a local dev server.

appcfg.py upload_data --application=dev~<APP_ID> --url=http://localhost:8080/_ah/remote_api --filename=<lengthyfilename>.backup_info
04:03 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20161230.160323
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20161230.160323.sql3
[ERROR   ] [Thread-1] RestoreThread:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\bulkloader.py", line 1555, in run
    self.PerformWork()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\bulkloader.py", line 2838, in PerformWork
    cursor.execute('select id, value from result')
DatabaseError: file is encrypted or is not a database

Inside the local admin page, there is not a restore option in any of the Cloud Datastore pages, I'm at a loss. All the answers I find are five years old, on outdated packages. How do you restore an Cloud Datastore Backup into a Developer App Server?

1
google-cloud-datastoregoogle-cloud-platformdatabase-backups
Are you using the gcloud tool as told in the official documentation? – Stubbies
No. My server is started using dev_appserver.py and I'm not aware that you can start an external datastore emulation when using the dev app server. – Savion Lee
Perhaps this helps? johanndutoit.net/… – GAEfan
@GAEfan I think i came across this before. If I use a folder name as it suggest, I get the error OperationalError: unable to open database file Clearly because it is not a file name. But if i give it anyone of the files in the backup I get this error DatabaseError: file is encrypted or is not a database. The latter error arises on the <filename>.backup_info files and the output-0` buried in the folders. – Savion Lee
@GAEfan I was able to download my most vital entities for testing using his method. (I tweeted to him and he updated the page!) Sadly though, the upload process that he provides only works with the download process also referenced by him. Not the copied backupfiles from the Datastore Admin. – Savion Lee

1 Answers

0
votes

One benefit of using the Datastore emulator is that it now supports import and export.

You can read about starting dev_appserver with the Datastore emulator here.