In GAE, how to make money transfer transaction idempotent?

Question

The GAE docs warn:

Whenever possible, make your datastore transactions idempotent so that if you repeat a transaction, the end result will be the same.

Suppose I want to transfer an amount of money between two people:

class User(ndb.Model):
    balance = ndb.IntegerProperty(default=0)

@ndb.transactional(xg=True)
def transfer(from_key, to_key, amount)
    from = from_key.get()
    to = to_key.get()
    from.balance -= amount
    to.balance += amount
    ndb.put_multi([from, to])

Since this isn't idempotent, it could happen more than once and cause a problem. I'd like to refactor this to ensure that the transaction is idempotent.

This answer suggests a solution:

To solve the problem, you can make the transaction idempotent by creating a "transaction Key" and recording that key in a new entity as part of the transaction. The second transaction can check for that transaction key, and if found, will do nothing. The transaction key can be deleted once you're satisfied that the transaction completed, or you give up retrying.

But I don't understand how to implement it.

Could someone explain how to make this transaction idempotent?

Have a read of this old article. Whilst it was written before a lot of new features existed in appengine, it is still worth looking at as a guide for implementing what you are trying to do. blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine — Tim Hoffman

snakecharmerb snakecharmerb · Accepted Answer · 2017-09-23T19:26:47

You could create a key based on the transaction details, for example:

import datetime
import hashlib

>>> txn = {
    'from_account': '100123',
    'to_account': '200456',
    'amount': 123456,
    'timestamp': datetime.datetime(2017, 9, 23, 10, 11, 12, 123456)
}

# Combine the values into a string
>>> raw_key = u''.join([unicode(v) for k, v in sorted(txn.items())])

>>> print raw_key
1234561001232017-09-23 10:11:12.123456200456

# hash the key so exposing it in logs etc. doesn't expose transaction data
>>> key = hashlib.sha256().hexdigest()
>>> print key
261c516faa580d6604850967c5804f3fce5f323aae90e36debdb84aa0b950dcb

You can store the hashed key in the datastore, or make it a computed property of your transaction model, if you have one, and query against it before attempting to create a new transaction.

class TransactionKeys(ndb.model):
        pass


class TransactionHandler(webapp2.RequestHandler):

   def post(self):
       txn = {
            'from_account': self.request.POST['from'],
            'to_account': self.request.POST['to'],
            'value': self.request.POST['value']
            'timestamp': datetime.datetime.now()
       }
       raw_key = u''.join([unicode(v) for k, v in sorted(txn.items())])
       txn_key = hashlib.sha256().hexdigest()
       ...
       transfer(from_key, to_key, amount, txn_key)


@ndb.transactional(xg=True)
def transfer(from_key, to_key, amount, txn_key)
    already_exists = TransactionKeys.get_by_id(txn_key)
    if already_exists:
        raise DuplicateTransactionError('Duplicate transaction!')
    else:
        transaction_key = TransactionKey(id=txn_key)
    from = from_key.get()
    to = to_key.get()
    from.balance -= amount
    to.balance += amount
    ndb.put_multi([from, to, txn_key])

This method is not perfect - it will fail if two identical transactions arrive in the same microsecond for example. You may be able to add other data to make the key more unique, such as the App Engine instance id or request id.

Finally, the obligatory disclaimer: I'm not a security professional, if you're doing this with real money you should perform appropriate levels of due diligence and consider professional indemnity/public liability insurance.

In GAE, how to make money transfer transaction idempotent?

1 Answers