Many different sharded counters in single transaction

Question

I have to increment three different counters in a single transaction. Beside that I have to manipulate three other entities as well. I get

too many entity groups in a single transaction

I've used the recipie from https://developers.google.com/appengine/articles/sharding_counters to implement my counters. I increment my counters inside some model (class) methods depending on business logic.

As a workaround I implemented a deferred increment method that uses tasks to update the counter. But that doesn't scale well if the number of counters increases further as there is a limit of tasks in a single transaction as well (I thinks it's 5) and I guess it's not the most effective way.

I also found https://github.com/DocSavage/sharded_counter/blob/master/counter.py which seems to ensure updating the counter even in case of a db error through memcache. But I don't want to increment my counters if the transaction fails.

Another idea is to remember the counters I have to increment during a web request and to increment them in a single deferred task. I don't know how to implement this in a clean and thread safe way without passing objects created in the request to the model methods. I think this code would be ugly and not in the same transcation:

def my_request_handler():
    counter_session = model.counter_session()
    model.mylogic(counter_session, other_params)
    counter_session.write()

Any experiences or ideas?

BTW: I'm using python, ndb and flask It would be ok if the counter is not 100% accurate.

Alex Dvoretsky Alex Dvoretsky · Accepted Answer · 2013-11-24T13:43:09

As said in Transactions and entity groups:

the simplest approach is to determine which entities you need to be able to process in the same transaction. Then, when you create those entities, place them in the same entity group by declaring them with a common ancestor. They will then all be in the same entity group and you will always be able to update and read them transactionally.

Many different sharded counters in single transaction

1 Answers