migration to HRD hung after 3 days also

Question

i am migrating my app to HRD, and it is about 3 days now. I also tried to do a pause and resume. It is still in the 'Copy' phase. I can see the change in Datastore read operations going up and getting billed.

I have also filed a issue at http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

I have not received the response for the issue yet. How long should i wait?

-Aswath

Welcome to StackOverflow. This is a programming question and answer site, not a vendor support site. If you have specific questions about programming, please ask them here. For information on what questions should and should not be asked here, please refer to the FAQ(http:/stackoverflow.com/faq). Thanks. :) — Ken White
Actually, Google has officially offloaded their AppEngine support to StackOverflow, so it is indeed a vendor support site now ;) — tdavis
@tdavis Only for coding questions. Production issues (of which this is one) should still go on the issue tracker. — Nick Johnson

klaaspieter klaaspieter · Accepted Answer · 2012-04-10T15:45:01

I had a similar problem when migrating my application. In the end it turned out I had several entities using more than the allowed number of indexes (I think it's 5000). Up until November 2011 App Engine was not enforcing the 5000 index entry / entity limit. This was fixed and all existing entity keys above the limit were grandfathered into the old behavior. The migration to HRD will change your keys so if you're exceeding the limit you'll have to decrease the number of entities.

I fixed my migration by removing the indexes which was possible because App Engine started using zig zag merge to merge the results if two separate queries. For example index:

- kind: Product
  properties:
  - name: name
  - name: tags
  - name: tags
  - name: tags

can be changed to:

- kind: Product
  properties:
  - name: name
  - name: tags

Other solutions suggested to me to fix the migration were:

imbalanced namespaces. If you use 5000 namespaces, but 99% of your entities are in 1 namespace, the tool shards by namespace and most of the entities will be serviced by a single worker instance. If you write new entities to this namespace faster than the single worker copies them to the new application, your migration will run forever
writing entities faster than the migration tool can map over keys. This is a typical case where if you are serving hundreds or thousands of queries per second (we call this QPS) and new entities are being written at that rate, the mapper cannot keep up. The mapper is what is responsible for sharding the entities into buckets so they can be copied in parallel

migration to HRD hung after 3 days also

1 Answers