Google App Engine GQL query question

Question

in theory, I want to have 100,000 entities in my User model:

I'd like to implement an ID property which increments with each new entity being put.

The goal is so that I can do queries like "get user with IDs from 200 to 300". Kind of like seperating the 10000 entities into 1000 readable pages of 100 entities each.

I heard that the ID property from App Engine does not guarantee that it goes up incrementally.

So how do I implement my own incrementing ID?

One of my ideas is to use memcache. Add a counter in memcache that increases each time a new entity is inserted.

class User(db.Model):
  nickname = db.StringProperty()
  my_id = db.IntegerProperty()


# Steps to add a new user entity:
# Step 1: check memcache
counter = memcache.get("global_counter")

# Step 2: increment counter
counter = counter + 1

# Step 3: add user entity
User(nickname="tommy",my_id=counter).put()

# Step 4: replace incremented counter
memcache.replace(key="global_counter",value=counter)

# todo: create a cron job which periodically takes the memcached global_counter and
# store it into the datastore as a backup ( in case memcache gets flushed )

what do you guys think?

additional question: if 10 users register at the same time, will it mess up the memcache counter?

Why do you require the ids to be sequential? How will you handle users wanting to have their accounts deleted? Won't that mess up your paging? — Mark Byers

James Polley James Polley · Accepted Answer · 2010-01-08T00:44:21

You don't need to implement your own auto-incrementing counter to achieve the pagination you're looking for - look at "Paging without a property"

In short, the key is guaranteed to be returned in a deterministic order, and you can use inequality operators (>=, <=) to return results starting from a particular key.

To get your first hundred users:

users = User.all().order("__key__").fetch(101)

The first 100 results are what you iterate over; the 101st result, if it is returned, is used as a bookmark in the next query - just add a .filter('__key__ >=', bookmark) to the above query, and you'll get results 200-301

Google App Engine GQL query question

2 Answers