Flask SQLAlchemy does not close MySQL database connections

Question

I have a Flask app using Flask-SQLAlchemy with a MySQL database where the db is defined as the following:

db.py:

from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()

main.py:

from db import db
app = Flask(__name__)
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = "mysql+pymysql://" + \
        DB_USERNAME + ":" + DB_PASSWORD + "@" + DB_HOST + "/" + DB_DATABASE
db.init_app(app)

@app.teardown_appcontext
def teardown_db(error):
    db.session.close()
    db.engine.dispose()

user.py:

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True, nullable=False)
    email = db.Column(db.String(120), unique=True, nullable=False)

I query my database using models using either db.engine.execute() to write raw SQL queries where required or use the integrated Flask-SQLAlchemy APIs for reading data such as User.query.filter_by().all().

I write new data into the db using the following:

new_user_entry = User(username = "abc", email = "[email protected]")
db.session.add(new_user_entry)
db.session.commit()

I am monitoring my MySQL server using show processlist and I notice that the database connections keep increasing by 2 for every single request that comes my way. The database connections seem to reset only when I stop the Flask process. With time, the MySQL server throws the below error:

`sqlalchemy.exc.TimeoutError: QueuePool limit of size 10 overflow 10 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)`

I am serving the app using gunicorn and gevent/eventlet with 2 worker processes. I use python3.

Am I missing something here? I tried ending the db session and disposing the engine, but this does not seem to work.

Answer 1

I finally found a fix to the above problem.

I used the declarative model defined in here instead of following the quickstart documentation for Flask-SQLAlchemy given here.

The changed files are as follows:

db.py:

from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base

engine = create_engine(DB_URI, convert_unicode=True)
db_session = scoped_session(sessionmaker(autocommit=False,
                                         autoflush=False,
                                         bind=engine))
Base = declarative_base()
Base.query = db_session.query_property()

def init_db():
    import user
    Base.metadata.create_all(bind=engine)

main.py:

from db import init_db, db_session

init_db()

@app.teardown_appcontext
def shutdown_session(exception=None):
    db_session.remove()

user.py:

from sqlalchemy import Column, Integer, String
from data_models.db import Base
class User(Base):
    id = db.Column(Integer, primary_key=True)
    username = db.Column(String(80), unique=True, nullable=False)
    email = db.Column(String(120), unique=True, nullable=False)

To query for records we could either use User.query.filter_by().all() or db_engine.execute().

To write new data into the database, we can use the following:

new_user_entry = User(username = "abc", email = "[email protected]")
db_session.add(new_user_entry)
db_session.commit()

Answer 2

In case we need to close session before creating a new child process (what is recommended), this is what we should use:

db.session.remove()
db.engine.dispose()

Like

from multiprocessing import Process
from app import db

@app.route('/process/start/', methods = ["GET"])
def process_start():
    db.session.remove()
    db.engine.dispose()
    p = Process(target = long_task)
    p.start()
    return 'started the long task'

def long_task():
    '''
    do long task
    '''