python - Google AppEngine: how to count a database's entries beyond 1000?

Question

Duplicate of "how does one get a count of rows in a datastore model in google appengine?"

---

I want to know how many users I have. Previously, I achieved this with the following code:

users = UserStore.all()
user_count = users.count()

But now I have more than 1,000 users and this method continues to return 1,000.

Is there an efficient programmatic way of knowing how many users I have?

Answer 1

It is indeed a duplicate and the other post describes how to theoretically do it, but I'd like to stress that you should really not be doing counts this way. The reason being that BigTable by its distributed nature is really bad for aggregates. What you probably want to do is add a transactional counter to that entity, and if there are lots of transactions a sharded counter. See: http://code.google.com/appengine/articles/sharding_counters.html

UPDATE: Since 1.3.1 cursors make stuff like this a lot easier: http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html#Query_Cursors

Answer 2

Use pagination like these examples here.

Answer 3

Since version 1.3.6 of the SDK the limit of 1000 on the count function has been removed. So a call to the count function will now return the exact number of entities, even if there are more than 1000. Only limitation would be if you had so many entities that the count function would not return before the request has a timeout.

Answer 4

For Python GAE SDK, you can increase the argument "limit" of the count method: https://developers.google.com/appengine/docs/python/datastore/queryclass#Query_count

Answer 5

I have write this method to count a query, but how said Nick Johnson maybe it's a bad idea...

def query_counter (q, cursor=None, limit=500):
  if cursor:
      q.with_cursor (cursor)
  count = q.count (limit=limit)
  if count == limit:
      return count + query_counter (q, q.cursor (), limit=limit)
  return count