Stop a background process in flask without creating zombie processes

Question

I need to start a long-running background process with subprocess when someone visits a particular view.

My code:

from flask import Flask
import subprocess

app = Flask(__name__)

@app.route("/")
def index():
    subprocess.Popen(["sleep", "10"])
    return "hi\n"

if __name__ == "__main__":
    app.run(debug=True)

This works great, for the most part.

The problem is that when the process (sleep) ends, ps -Af | grep sleep shows it as [sleep] <defunct>.

From what I've read, this is because I still have a reference to the process in flask.

Is there a way to drop this reference after the process exits?

I tried doing g.subprocess = subprocess.Popen(["sleep", "10"]), and waiting for the process to end in @app.after_request(response) so I can use del on it, but this prevents flask from returning the response until the subprocess exits - I need it to return the response before the subprocess exits.

Note:

I need the subprocess.Popen operation to be non-blocking - this is important.

Try extend Thread class and implement your code there... And to close it, use join https://docs.python.org/2/library/threading.html#threading.Thread.join — Valijon
Try to rework your project to use Celery (celeryproject.org) or something else that has a robust way of doing background processing instead. — jsbueno

jsbueno jsbueno · Accepted Answer · 2017-01-03T17:14:56

As I've suggested in the comments, one of the cleanest and most robust way of achieving this kind of thing in Python is by using celery.

Celery requires a broker transport for messaging, for which rabbitmq is the default, and at least a process with workers running. However, the thing that increases readbility an dmaintanability is that the worker code can co-exist in the same file or files than your server app. You invoke the remote procedures as though it where a simple function call.

Celery can handle retries, post-task events, and lots of other things for free, everything with mature code hardened by years of use in production.

This is your example after re-writting it for use with Celery:

from flask import Flask
from celery import Celery
import subprocess

app = Flask(__name__)
celery_app = Celery("test")

@celery_app.task
def run_process():
    subprocess.Popen(["sleep", "5"])

@app.route("/")
def index():
    run_process.delay()
    return "hi\n"

if __name__ == "__main__":
    app.run(debug=True, port=8080)

With this code, in a system with the rabbitmq server running with default options (I installed the package, and started the service - no configurations whatsoever. Of course on production you would have to tune that - but if everything is to be on the same server, it may not even be needed.)

With rabbitmq in place, one starts the worker process with a command line like: celery worker -A bla1.celery_app -D (pip install celery on the same virtualenv you have your Flask). Then just launch the flask server and see it working.

Of course this has even more advantages if you are doing more work in Python itself than just calling an external process. It can have access to your database models, and you can perform assynchronous actions that modify objects in there (and eventually trigger responses for the user, as "flash" messages on the user session, or e-mails)

Stop a background process in flask without creating zombie processes

2 Answers