Flask/Werkzeug debugger, process model, and initialization code

Question

I'm writing a Python web application using Flask. My application establishes a connection to another server at startup, and communicates with that server periodically in the background.

If I don't use Flask's builtin debugger (invoking app.run with debug=False), no problem.

If I do use the builtin debugger (invoking app.run with debug=True), Flask starts a second Python process with the same code. It's the child process that ends up listening for HTTP connections and generally behaving as my application is supposed to, and I presume the parent is just there to watch over it when the debugger kicks in.

However, this wreaks havoc with my startup code, which runs in both processes; I end up with 2 connections to the external server, 2 processes logging to the same logfile, and in general, they trip over each other.

I presume that I should not be doing real work before the call to app.run(), but where should I put this initialization code (which I only want to run once per Flask process group, regardless of the debugger mode, but which needs to run at startup and independent of client requests)?

I found this question about "Flask auto-reload and long-running thread" which is somewhat related, but somewhat different, and the answer didn't help me. (I too have a separate long-running thread marked as a daemon thread, but it is killed when the reloader kicks in, but the problem I'm trying to solve is before any reload needs to happen. I'm not concerned with the reload; I'm concerned with the extra process, and the right way to avoid executing unnecessary code in the parent process.)

Thanks for this question (and answer), this one had me puzzled when trying to something similar. — akatkinson

metamatt metamatt · Accepted Answer · 2012-07-20T06:09:49

I confirmed this behavior is due to Werkzeug, not Flask proper, and it is related to the reloader. You can see this in Werkzeug's serving.py -- in run_simple(), if use_reloader is true, it invokes make_server via a helper function run_with_reloader() / restart_with_reloader() which does a subprocess.call(sys.executable), after setting an environment variable WERKZEUG_RUN_MAIN in the environment which will be inherited by the subprocess.

I worked around it with a fairly ugly hack: in my main function, before creating the wsgi application object and calling app.run(), I look for WERKZEUG_RUN_MAIN:

if use_reloader and not os.environ.get('WERKZEUG_RUN_MAIN'):
    logger.warning('startup: pid %d is the werkzeug reloader' % os.getpid())
else:
    logger.warning('startup: pid %d is the active werkzeug' % os.getpid()
    # my real init code is invoked from here

I have a feeling this would be better done from inside the application object, if there's a method that's called before Werkzeug starts serving it. I don't know of such a method, though.

This all boils down to: in Werkzeug's run_simple.py, there's only going to be one eventual call to make_server().serve_forever(), but there may be two calls to run_simple() (and the entire call stack up to that point) before we make it to make_server().

Flask/Werkzeug debugger, process model, and initialization code

1 Answers