Gunicorn, Django, Gevent: Spawned threads are blocking

Question

we recently switched to Gunicorn using the gevent worker.

On our website, we have a few tasks that take a while to do. Longer than 30 seconds.

Preamble

We did the whole celery thing already, but these tasks are run so rarely that its just not feasible to keep celery and redis running all the time. We just do not want that. We also do not want to start celery and redis on demand. We want to get rid of it. (I'm sorry for this, but I want to prevent answers that go like: "Why dont you use celery, it's great!")

The tasks we want to run asynchronously

I'm talking about tasks that perform 3000 SQL queries (inserts) that have to be performed one after each other. This is not done all too often. We limited to running only 2 of these tasks at once as well. They should take like 2-3 minutes.

The approach

Now, what we are doing now is taking advantage of the gevent worker and gevent.spawn the task and return the response.

The problem

I found that the spawned threads are actually blocking. As soon as the response is returned, the task starts running and no other requests get processed until the task stops running. The task will be killed after 30s, the gunicorn timeout. In order to prevent that, I use time.sleep() after every other SQL query, so the server gets a chance to respond to requests, but I dont feel like this is the point.

The setup

We run gunicorn, django and use gevent. The behaviour described occurs in my dev environment and using 1 gevent worker. In production, we will also run only 1 worker (for now). Also, running 2 workers did not seem to help in serving more requests while a task was blocking.

TLDR

We consider it feasible to use a gevent thread for our 2 minute task (over celery)
We use gunicorn with gevent and wonder why a thread spawned with gevent.spawn is blocking
Is the blocking intended or is our setup wrong?

Thank you!

Just executing code in a greenlet doesn't make the code non-blocking . You have to actually be making calls to asynchronous APIs for the greenlet to not block. The calls you're making to time.sleep will still block inside a greenlet, for example. You should use gevent.sleep to do a non-blocking sleep. Your database calls are probably blocking too, unless you're using gevent monkey patching. — dano
@dano As I'm using gunicorn with a gevent worker, the monkey patching is taken care of for me. time.sleep allows for a thread change to happen, so it is non blocking. Maybe monkey patching patched it. But, I expected to hand the task to a worker who then takes care of it. So it can be blocking all it wants as long as its run in parallel. But I guess greenlets cannot be run in parallel? — enpenax
greenlets can be run concurrently, but they are still single-threaded; only one of them can be using the CPU at a time. So, if one greenlet is in a blocking I/O call or gevent.sleep, another greenlet can run. But if one greenlet is crunching numbers or parsing XML (or any other CPU-based operation), no other greenlets will be running. A greenlet will also block other greenlets if it's do an I/O operation that isn't asynchronous - meaning it's not monkey patched by gevent or otherwise plugged into gevent's event loop. — dano
You say you're doing inserts, but later down say your using django ORMs. So you're probably doing more than inserts (otherwise you could do that 3000 insert in one sql call). Since it's using CPU it will block, it's as simple as that. Stop this whining about not wanting to run celery and redis. Redis is lightweight and you can have celery on autoscale from 0 to 1. Chances are you will use redis for a lot more. — dalore
@dalore and sometimes you are not the master over these decisions are are forced to roll with something else. plus it was nowhere stated I would be using Django ORM. my solution below is running fine for months now but thank you for your time — enpenax

johntellsall johntellsall · Accepted Answer · 2014-07-02T06:13:53

One way to run a task in the background is to fork the parent process. Unlike with Gevent, it won't block -- you're running two completely separate processes. This is slower than starting another (very cheap) greenlet, but in this case it's a good trade off.

Your process splits into two parts, the parent and the child. In the parent, return a response to Gunicorn just like in normal code.

In the child, do your long-running processing. At the end, clean up by doing a specialized version of exit. Here's some code that does processing and sends emails:

    if os.fork():
        return JsonResponse({}) # async parent: return HTTP 200
    # child: do stuff, exit quietly
    ret = do_tag_notify(
        event, emails=emails, photo_names=photo_names,
        )
    logging.info('do_tag_notify/async result={0}'.format(ret))
    os._exit(0)             # pylint: disable=W0212
    logging.error("async child didn't _exit correctly") # never happens

Be careful with this. If there's an exception thrown in the child, even a syntax error or unused variable, you'll never know about it! The parent with its logging is already gone. Be verbose with logging, and don't do too much.

Using fork is a useful tool -- have fun!