App Engine: Mechanics of creating a unique entity if it doesn't exist

Question

We have an issue where we want to lazily create an entity if it does not exist. There is some discussion going on about how to do this and I would like to clarify some things around app engine transactions. I will limit my query to single entity group transactions.

I am using Go in my examples, but I hope the code is clear enough for non-Go programmers.

My understanding is that a transaction, on a single entity group, will succeed only if the entity group is not modified externally during the transaction. The 'entity group timestamp' indicating when an entity group was changed is stored in the root entity of the entity group. So during a transaction the current 'entity group timestamp' is read and the transaction can only succeed if it hasn't changed by the end of the transaction.

key := datastore.NewKey(c, "Counter", "mycounter", 0, nil)
count := new(Counter)
err := datastore.RunInTransaction(c, func(c appengine.Context) error {
  err := datastore.Get(c, key, count)
  if err != nil && err != datastore.ErrNoSuchEntity {
    return err
  }
  count.Count++
  _, err = datastore.Put(c, key, count)
  return err
}, nil)

In the example above (taken from https://cloud.google.com/appengine/docs/go/datastore/transactions) there are two non-error cases, I can see:

The Get succeeds and the 'entity group timestamp' on the counter can be used to ensure no other transactions update the counter during this transaction.
The Get fails with ErrNoSuchEntity and the Put is used to store the counter for the first time.

In the second case it is possible that another identical transaction is running. If both transactions' Get return ErrNoSuchEntity how does the datastore ensure that only one put succeeds? I would expect there to be no "entity group timestamp" in the datastore to test against?

Does the transaction know that it needs to test for the non-existence of the counter in order for the Put and the entire transaction to succeed?

Is there a chance in this case for two transactions to succeed and for one Put to overwrite the other?

If there is documentation, or videos etc, around the mechanism that controls this I would love to read it.

This description of a transaction might help clarify my question. It lays out, more clearly than I have, the user of 'timestamp's in app engine transactions. "When a transaction starts, App Engine uses optimistic concurrency control by checking the last update time for the entity groups used in the transaction. Upon commiting a transaction for the entity groups, App Engine again checks the last update time for the entity groups used in the transaction. If it has changed since our initial check, an error is returned." cloud.google.com/appengine/docs/go/datastore/transactions — Francis Stephens
Jaime and Clément are right. when the two txes commit and recheck the entity group, one will find no timestamp and commit successfully, the other will find the first's timestamp and fail. the datastore API code will then retry it. — ryan
background: paper on megastore, the storage system that powers the datastore. another description of datastore transactions, start on slide 49. source code underlying python NDB's get_or_insert, which implements this use case. — ryan

Clément Clément · Accepted Answer · 2015-05-26T21:35:51

From the documentation on transactions:

Inside of transactions, serializable isolation is enforced.

I suggest reading up a bit on the linked wikipedia page, but in brief, the datastore will make sure that the final outcome is as if the two transactions ran in sequence.

Having both transactions write a new, zeroed, counter is not a possible outcome.

Does the transaction know that it needs to test for the non-existence of the counter in order for the Put and the entire transaction to succeed?

In a way yes: the transaction can succeed on first try only if there was no overlapping transaction touching the same key.

Is there a chance in this case for two transactions to succeed and for one Put to overwrite the other?

No, if the two transactions timeframes overlap, then the last one to commit will fail, be eventually retried and then will see the existing counter and increment it.

App Engine: Mechanics of creating a unique entity if it doesn't exist

2 Answers