Why does multiprocessing work on Django runserver and not on ngnix uwsgi?

Question

I have a Django 1.6 using python3.3 application which receives an http request, does short term work, starts a new process and returns in a matter of 2 seconds. The process typically takes 50-60 seconds at which time it writes that the data is free on the data base where a timed ajax call can retrieve the data and update the clint web page.

This works perfectly on the Django development runserver.

When I deploy the application on ngnix uwsgi the quick response is delayed for 50-60 seconds and appears in tandem with the completion of the process.

I logged the time between entering the view and just before sending the response and it was 1-2 seconds.

I checked the packets with wireshark and all communication ceases for the duration of the process i.e 50-60 seconds.

in the uwsgi .ini file I used processes=3

I would be grateful for a solution or a line of investigation.

the .ini file:

[uwsgi], chdir=/nnnhn/, module=wsgi, #master=True, pidfile=/tmp/project-master.pid, vacuum=True, max-requests=5000, daemonize=/var/log/uwsgi.log, socket=/tmp/uwsgi.sock, uid=hhm, processes=3, pythonpath=/csghgfh/doo

the process spawning code:

process=multiprocessing.Process(target=Util.processImage1, args=(processUtil, img_full_path, snapshot.pk, process_pk, db_name, process_table_name))
process.start()

the relevant spawned code:

def processImage1(self, img_full_path, snapshot_pk, process_pk, db_name, process_table_name):
    connection.close()
    print('')
    print('I think I just closed th db connection connection')
    print('')
    print('processImage1:', datetime.now())
    print('process_pk: ', process_pk)
    sys.stdout.flush()

    try:
        con = psycopg2.connect(host = 'a', database=db_name, user='a') 
        cur = con.cursor()

you mind adding your uwsgi.ini and the code that spawns the proc? — Tommaso Barbugli

roberto roberto · Accepted Answer · 2014-05-29T10:25:51

This is explained in the (probably) most important page of the uWSGI docs: http://uwsgi-docs.readthedocs.org/en/latest/ThingsToKnow.html

When you fork() the process, the new one will inherit the connection descriptor. You have to close it in the new forked process, or you can add the --close-on-exec option to force it automatically

Why does multiprocessing work on Django runserver and not on ngnix uwsgi?

2 Answers