Task Queues, Idempotence, and Objectify Transactions

Question

The documentation for GAE's Task Queue API states:

You can enqueue a task as part of a datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully.

However, the documentation for datastore transactions states twice that we should make them idempotent whenever possible, and submitting to a task queue is not idempotent. The documentation for objectify takes this a step further, explaining that work MUST be idempotent within its transactions.

So, is there a standard way to handle combining these recommendations/requirements, or should I roll my own technique (perhaps using something like this)?

Answer 1

There is also the concern that a task can execute twice (or more) - the queue provides "at least once" semantics not "exactly once" semantics. This is common.

Some operations are easy to make idempotent (eg, "set birthdate"). Some operations can be difficult to make idempotent (eg, "transfer $5 from account A to account B"). For the difficult ones, usually the trick involves creating a transaction id outside of the start of the transaction sequence and making sure that id follows the whole chain, even through tasks. If anything retries and sees the transaction id has already been completed, you can just return.

Answer 2

If the task was enqueued than anything else in its associated transaction was committed as well. Yes, technically it is possible for a transaction to be committed and still get an error response (e.g. a timed-out accepting the successful response) though that is not common. In any case your task should be idempotent as well (it could use the data committed in its own transaction to help with that) as the task could be executed more than once even if you submitted it once. see Why Google App Engine Tasks can spuriously be executed more than once?.