atomic counter increment in App-Engine

Question

I am still a bit confused about transactions, whether it is using DatastoreService or Objectify. (Yes, I read What is the correct way to atomically increment a counter in App Engine?). I need to increment a counter atomically. How do I do that? The example in the app engine docs has a rollback in its finally block. But I don’t want a rollback, I want the system to keep trying. On the other hand, the objectify docs say that its transaction model is different from that of the low-level api. So I am writing both codes, I just need help correcting them or confirming them.

DatastoreService version

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService()
Transaction txn = datastore.beginTransaction();
try {
    Key commentKey = KeyFactory.createKey(“Comment”, id);
    Entity comment = datastore.get(commentKey);
    int views = (Integer)comment.getProperty(“views”);
    views++;//increment step
    comment.setProperty(“views”, views);

    datastore.put(comment);

    txn.commit();
} finally {
    if (txn.isActive()) {
        txn.rollback();
    }
}

Objectify version

ofy().transact(new VoidWork() {

      @Override
      public void vrun() {
        Comment comment = ofy().load().type(Comment.class).id(commentId).now();
        long views = 1+ comment.getViews();
        comment.setViews(views);
        ofy().save().entity(comment).now();
      }
    });

An important point is that I want the system to keep trying ad infinitum. And of course I want the client call to return while all of this happens asynchronously. Thanks for any help

scope of the question is too broad if you include "try forever" as its complex needing messaging from task queues — Zig Mandel
By forever, I mean I really want the counter to be extremely accurate, whereas in the DatastoreService version there is a rollback and I am not sure how many times it tried. — Katedral Pillon
appengine frontend has a 1 minute limit so at that point its complex as you need to handle many cases — Zig Mandel
you should look at how to combine with memcached which has a robust atomic inctement. but too broad to put here. other s.o. answers mention this. — Zig Mandel
@ZigMandel so are my two answers incorrect? What are they missing? I am particularly interested in objectify, is it correct for best effort? — Katedral Pillon

stickfigure stickfigure · Accepted Answer · 2015-10-06T06:12:26

The Objectify version will retry if there is a concurrency collision. You could modify the DatastoreService version to loop on ConcurrentModificationException and you will get effectively the same logic.

This, however, does not get you an extremely accurate counter (although it's usually close enough for most purposes). You wouldn't, however, want to run bank transactions like this.

The problem (present in all distributed transaction processing systems) is that something could go wrong during your transaction, throwing say, DatastoreException. This leaves your counter in an indeterminate state - did the commit succeed or not? You don't know.

If you want to be exact (and you did say extremely accurate), you need to perform some variation of this:

Create a txn record with a unique key before you start the transaction
Start a transaction
Check to see if the record exists; if not, you're already done
Remove the record, increment the counter, and commit
If error, repeat from 2 until success

And you'll need some sort of query to clean up leftover txn records for transactions that completely failed.

A variation of this is to create just the id before the transaction and create the txn record inside the transaction, using the positive existence of the record to indicate success. If you keep that record around long-term, it's basically a transaction history.

This level of transactional certainty costs additional write operations and adds noticeable delay, so you probably only want to use it when you really need accuracy.

atomic counter increment in App-Engine

1 Answers