Google Datastore transaction exception

Question

As noted in the Google Datastore documentation:

If your application receives an exception when committing a transaction, it does not always mean that the transaction failed. You can receive errors in cases where transactions have been committed.

In addition to updating an entity within the transaction, I am also adding push tasks to a push queue. If the transaction throws an exception, but has really been committed successfully, does this mean that the push queue tasks were enqueued successfully?

Answer 1

If the Transaction was indeed successful - even if it faced exceptions - the push queue tasks should be enqueued correctly as well.

As per the documentation that you mentioned here, it says:

You can enqueue a task as part of a Datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. Tasks added in a transaction are considered to be a part of it and have the same level of isolation and consistency.

This means that this push will occurs with the rest of the transaction, using the same functions and features from transactions. This way, if the transaction was successful, you should have your tasks enqueued correctly.

Let me know if the infor

Answer 2

From the quote in gso_gabriel's post the answer appears to be yes.

However I have some doubts. The transactional task enqueueing appears to be a feature of the datastore client library (ndb/your java library) - I don't remember seeing it documented as a generic datastore (i.e. server-side) feature.

If that's correct then the answer may be dependent on the actual library implementation. This is where my doubts come from - in some cases the client cannot determine from the exception itself if the transaction was successfully committed or not. For example see app engine datastore transaction exception - though the answer also suggests that the task enqueueing would be handled correctly server-side). It might be possible to still ensure correct task enqueueing with additional logic, but again - implementation-dependent.

If you want to avoid such uncertainty altogether I believe you have 2 options:

• if it's important to have the task enqueued correctly and in a timely manner even in these adverse conditions you could try to use task naming as means of making the task enqueueing idempotent: even if the exception can't indicate if the task was enqueued correctly or not retrying won't hurt as you cannot (for a while) enqueue a named task twice. You'll need a way of generating unique task names, though.

An advantage of assigning your own task names is that named tasks are de-duplicated, which means you can use task names to guarantee that a task is only added once. De-duplication continues for 9 days after the task is completed or deleted.

• if the timing of executing the task is not that important it may be possible to deal with this whole story in a different manner. For example I already had in place some periodic higher-level integrity check jobs looking for orphaned items (regardless of the reasons for their existence). In my case the effects of a missed task enqueueing was just one way of producing such orphan entity. Since timing was not crucial and the chances of encountering such events are low it was acceptable to not even worry about them, keep things simple and leave recovery (re-trying the task enqueuing) to these integrity check jobs.