HRD migration broke datastore queries & indexes

2
votes

Google appengine HRD migration has been a nightmare for me. I migrated my 55GB datastore to HRD yesterday. Since then many queries and indexes are broken:

Some examples:

  • Select * from table1 where col1=val1 => query.get() returns empty in python. However, it works in datastore viewer.
  • Select * from table1 where col1=val1 => query.count()>0. However query.get() = empty.
  • Select * from table1 where col1=val1 order by col2 desc => Almost half of the rows are getting missed in the response. Same behavior in datastore viewer.

How do I get these tables and indexes repaired? Any way of getting Google Appengine team support for addressing this issue? Its a GAE Migration tool bug.

Will appreciate any help.

3
google-app-enginemigrationdatabase-migration

3 Answers

4
votes

When the migration tool is used, a new app id is assigned, which makes all the keys change.

To recreate the custom indexes:

  1. Temporarily empty index.yaml.
  2. Vacuum the indexes (check out How can I remove unused indexes in Google Application Engine? for further information).
  3. Wait until all the indexes have been deleted.
  4. Restore index.yaml.
  5. Create indexes by either redeploying the application or running appcfg.py update_indexes <path> (check out the documentation for furher information).

You may also need to manually update all of the other references (e.g. a ListProperty of keys) if you have any.

Edit
The simple, mono-property indexes that are managed automatically by App Engine are created/updated when a property is put.
To regenerate them, I recommend creating and running a simple MapReduce task to put every existing entity. This procedure should rebuild all the indexes (including those defined in index.yaml).
As this is a costly process, first do it manually with a few entities to see if it solves the problem.

1
votes

Tables get repaired automatically in about 2-3 days. Thats a HRD problem. My problem is now resolved.

0
votes

Update : finally it fix itself in 24 hours=)

I have the same problem as you

query.count()>0. However query.get() or fetch() is empty.

Its strange that some table work fine but some table has this problem

I think it is Google App Engine problem from migration very large table(model).

I hope my tables will recovered like yours in 2-3 days too.