GAE sync production and dev data-store

Question

I am able to download a backup of my production GAE data-store but I am not able to upload the backup to my dev app. Here are my attempts:

Downloading store from production, works as expected:

appcfg.py download_data --application=s~myApp --url=http://myApp.appspot.com/_ah/remote_api --filename=backup1

Upoading backup to dev store, fails:

appcfg.py upload_data --application=dev~myApp  --url=http://localhost:8080/_ah/remote_api --filename=backup1
Uploading data records.
[INFO    ] Logging to bulkloader-log-20120823.225047
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20120823.225047.sql3
Please enter login credentials for localhost
Email: [email protected]
Password for [email protected]: 
[INFO    ] Connecting to localhost:8080/_ah/remote_api
[INFO    ] Starting import; maximum 10 entities per post
............................................................................................................................................................[INFO    ] Unexpected thread death: WorkerThread-5
[INFO    ] An error occurred. Shutting down...
.........[ERROR   ] Error in WorkerThread-5: <urlopen error [Errno 61] Connection refused>

[INFO    ] 2670 entities total, 0 previously transferred
[INFO    ] 1650 entities (1207416 bytes) transferred in 431.1 seconds
[INFO    ] Some entities not successfully transferred

It seems error 61 is a port error. The app is running, I've signed in as admin via the browser and I've also put in my email/password as part of the CLI command. Some records always get uploaded and then the upload process gets cut off.

What credentials must I use? I enter the ones I have but it doesn't work. using SDK1.9.9 — Tjorriemorrie

Tim Hoffman Tim Hoffman · Accepted Answer · 2012-08-25T00:25:33

I have always found uploading data to the local data store more reliable setting the number of threads to 1. Rememeber the dev server is essentially single threaded. Try running it with --num_threads=1.

I would also recommend setting --db_filename=DB_FILENAME to create a progress db, which means you can restart from where it got to if it fails.

I have also at times had to reduce the batch size when trying to write large entities.

Cheers

GAE sync production and dev data-store

1 Answers