download_data and upload_data keys differences in AppEngine

Question

According to AppEngine documentation (quoted below), using download_data and upload_data should preserve the original keys of the downloaded data.

When data is downloaded, the entities are stored along with their original keys. When the data is uploaded, the original keys are used. If an entity exists in the datastore with the same key as an entity being uploaded, the entity in the datastore is replaced.

I am using these two commands:

appcfg.py download_data --oauth2 --url=https://myapp-id.appspot.com/_ah/remote_api --filename=testdata/gamedata

appcfg.py upload_data --url=http://localhost:9876/_ah/remote_api  --filename=testdata/gamedata

The data uploads with same ids but the keys are different, any idea what I am doing wrong? Or is the keys are computed differently il the local devappserver?

Update

Adding some data to compare between the prod and the local dev version

Production key: ag9zfmF3ZXNvbWVzdGdhbWVyGAsSCENhdGVnb3J5IgrYo9is2YbYqNmKDA

After download_data from PROD and upload_data to localhost, the key turns into for that same object: ahFkZXZ-YXdlc29tZXN0Z2FtZXIYCxIIQ2F0ZWdvcnkiCtij2KzZhtio2YoM

It would help if you showed some of the data you observe and your model classes. — Guido van Rossum

crazystick crazystick · Accepted Answer · 2013-06-25T18:20:38

You need to use the appropriate transforms in your bulkloader config, for example if you are using ids, then something like

- kind: YourEntity
  connector: csv
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string
      import_transform: transform.none_if_empty(transform.create_foreign_key('YourEntity', key_is_id=True))

    - property: more props...

would transform the keys on the way in and on the way out.

If you have entities with parents, then you need to split on the way out and recombine on the way in:

- kind: ChildEntity
  connector: csv

  property_map:
    - property: __key__
      external_name: childKey
      import_transform: transform.create_deep_key(('ParentEntity', 'parentKey', True),('ChildEntity', transform.CURRENT_PROPERTY, True))

      export:
        - external_name: parentKey
          export_transform: transform.key_id_or_name_as_string_n(0)

        - external_name: childKey
          export_transform: transform.key_id_or_name_as_string_n(1)

download_data and upload_data keys differences in AppEngine

1 Answers