fail to insert data into local Google App Engine datastore

Question

I am following the example from Google's Mobile Shopping Assistant sample which asks me to import data according to this link.

I tried the steps according to the example (all the sample codes are vanilla, i didn't change any thing except to fix the warning to use the latest Gradle build version)

I believe that I am missing an essential step that is not stated in the example. Can someone provide some insights into which steps I did wrongly?

the following are the steps i did

start local googleAppEngine app "backend"
ran cmd " appcfg.py upload_data --config_file bulkloader.yaml --url=http ://localhost:8080/remote_api --filename places.csv --kind=Place -e [email protected]".

This command is supposed to insert 2 rows into the datastore (places.csv contains two entries)
this gave me the following readout

10:07 AM Uploading data records.

[INFO    ] Logging to bulkloader-log-20151020.100728

[INFO    ] Throttling transfers:

[INFO    ] Bandwidth: 250000 bytes/second

[INFO    ] HTTP connections: 8/second

[INFO    ] Entities inserted/fetched/modified: 20/second

[INFO    ] Batch Size: 10

Error 302: --- begin server output ---

--- end server output ---

I then go to "http://localhost:8080/admin/buildsearchindex" which displays "MaintenanceTasks completed."
Next I go to "http://localhost:8080/_ah/admin" but it displays

Datastore has no entities in the Empty namespace. You need to add data

programatically before you can use this tool to view and edit it.

@brandall unfortunately no, I skipped that and followed the example from rominirani.com/2014/01/10/… instead. — Angel Koh
Thanks for your reply. I'll take a look at the link. So frustrating, I cannot see anything I'm doing wrong.... :( — brandall
The 'Error 302' in the response to the bulkloader seems to indicate that the insert failed. Perhaps taking a look at the server output / bulkoader log will give a clue as to what happened? — Adam
@Adam the log shows the same as above (finishes with 'end server output'). I think I'm going to add a bounty to this question, as it's too basic an issue to be left unanswered! — brandall

stephanmunich stephanmunich · Accepted Answer · 2016-01-08T10:12:35

I had the same problem but not with the local developer server but with the deployed version. After nearly going insane, I found a workaround to upload the data using appcfg. In my case, I noticed that when trying the following

Input not working for me:

gcloud auth login
appcfg.py upload_data --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<[email protected]>

Output of error:

11:10 AM Uploading data records.
[INFO    ] Logging to bulkloader-log-20160108.111007
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Error 302: --- begin server output ---

--- end server output ---

as expected, I was not asked to authenticate myself again during the second command but apparently, appcfg could still not authenticate my account. I am using Win7, with Python 2.7, the Python Google App Engine SDK including appcfg.py and gcloud from the Google Cloud SDK if I get it right.

However, on https://cloud.google.com/container-registry/docs/auth it is shown that you can print out the access token using the gcloud command and then insert it manually in your appcfg command which worked for me

Input working for me:

gcloud auth login
gcloud auth print-access-token

This prints out the access token which you can use with appcfg

appcfg.py upload_data --oauth2_access_token=<oauth2_access_token> --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<[email protected]>

Output of successful data upload:

10:42 AM Uploading data records.
[INFO    ] Logging to bulkloader-log-20160108.104246
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20160108.104246.sql3
[INFO    ] Connecting to <yourproject>.appspot.com/remote_api
[INFO    ] Starting import; maximum 10 entities per post
.
[INFO    ] 3 entities total, 0 previously transferred
[INFO    ] 3 entities (4099 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

I hope this helps anybody trying to solve this problem. To me, it is not clear what the source of this problem is, but this is a workaround that works for me.

BTW, I observed the same problem on a mac.

fail to insert data into local Google App Engine datastore

3 Answers