Django save() behavior with autocommit transactions

8
votes

I have a following setup:

  • Several data processing workers get configuration from django view get_conf() by http.
  • Configuration is stored in django model using MySQL / InnoDB backend
  • Configuration model has overridden save() method which tells workers to reload configuration

I have noticed that sometimes the workers do not receive the changed configuration correctly. In particular, when the conf reload time was shorter than usual, the workers got "old" configuration from get_conf() (missing the most recent change). The transaction model used in Django is the default autocommit.

I have come up with the following possible scenario that could cause the behavior:

  1. New configuration is saved
  2. save() returns but MySQL / InnoDB is still processing the (auto)commit
  3. Workers are booted and make http request for new configuration
  4. MySQL (auto)commit finishes

Is the step 2 in the above scenario possible? That is, can django model save() return before the data is actually committed in the DB if the autocommit transactional method is being used? Or, to go one layer down, can MySQL autocommitting INSERT or UPDATE operation finish before the commit is complete (update / insert visible to other transactions)?

2
mysqldjangodjango-modelsinnodb
Are you using InnoDB or MyISAM for your engine? - FlipperPA
InnoDB. The DB is running on Amazon RDS with default configuration. There are some large tables, but the table(s) related to this problem are small (on the order of 128kb or so) - jhonkola
Can you turn off autocommit just for this case? - sobolevn
Can you provide a little more info on the mechanics of "tells workers to reload"? If the workers are in a different process (e.g., if you are notifying via Celery or Python-RQ), then yes - your step 2 can/will almost definitely happen. - bimsapi
Modifying my comment, a little: on databases I'm familiar with (PostgreSQL, MSSQL, Oracle) commits of any kind (auto or manual) block until complete. So, the flow you describe should work, since the db call happens before the signal. If the entire view were transactional (e.g., ATOMIC_REQUESTS=True, then you could have a race where the other process tries to load before the commit occurs). Considering the workers get their conf via an HTTP connection, is there caching occurring at any layer? - bimsapi

2 Answers

0
votes

Object may be getting dirty, please try refresh object after save.

obj.save()

obj.refresh_from_db()

reference: https://docs.djangoproject.com/en/1.8/ref/models/instances/#refreshing-objects-from-database

0
votes

This definitely looks like a race condition.

The scenario you describe should never happen if there's only one script and one database. When you save(), the method doesn't return until the data is actually commited to the database.

If however you're using a master/slave configuration, you could be the victim of the replication delay: if you write on the master but read on the slaves, then it is entirely possible that your script doesn't wait long enough for the replication to occur, and you read the old conf from the slave before it had the opportunity to replicate the master.

Such a configuration can be set up in django using database routers, or it can be done on the DB side by using a DB proxy. Check that out.