Error retrieving and writing entities in Google App Engine's remote_api_shell.py

Question

Here's my dilemma: I've got to run through a list of entities checking if some repeated property--I'm using NDB--is empty. If it is, then I assign a value to it and put(). Otherwise I skip the entity. I'm trying to do all of this in the remote login shell that comes with google app engine.

I've tried just iterating over Model.query(), doing the conditional, and writing the value, but when I start to write the process gets hung up. When I finally Ctrl-C, it pops out an error saying, "assert response.set_status_size() == len(server_keys); AssertionError". I'm assuming this has something to do with the size of the entities it's trying to retrieve. Anyone know what's up? Here's my current code:

>>> for entity in Model.query():
...    if not len(entity.references):
...        entity.references = somevalue
...        continue
...    print 'skipped'

I would just filter the query instead of using the if statement but I'm not sure how to filter a query by the length of a repeated property.

Answer 1

How many entities do you have? If you have more than 100 or so, querying from remote api is really inefficient (quadratic). So that may explain that it hangs.

You are right that you cannot write a query that detects the condition you are looking for.

Answer 2

The code sample returns when it finds the first entity with no references (or prints skipped if there are none such) but it may leave some entities without any references. I don't know if that's what you intended.

If only a few values of references are possible, eg ['python', 'ruby', 'php'], I think you could do a Repeated Properties query like Model.query(Model.references.IN(['python', 'ruby', 'php']))), take the difference of that list with the list of all entities, and iterate over remaining entities. (I don't know much NDB, and doubt this would be a good method in general, but expect it would work ok for some data sets.)

Answer 3

I finally got it working! I used query.fetch_page to fetch the query in batches, and then iterated over each batch and applied the changes.

>>> more = True
>>> cursor = None
>>> while more:
...     batch, cursor, more = MyModel.query().fetch_page(50, start_cursor=cursor)
...     for entity in batch:
...          if not len(entity.references):
...               entity.references = somevalue
...               entity.put()
...     print "finished batch"