Using both gevent (or eventlet) and prefork workers with Celery

Question

Within the Concurrency section of the Celery docs it states that:

...mix of both Eventlet and prefork workers, and route tasks according to compatibility or what works best

Source: http://celery.readthedocs.org/en/latest/userguide/concurrency/eventlet.html#concurrency-eventlet

This means that it's possible to have a worker use the gevent/eventlet pool implementation while another uses a prefork pool.

The pool implementation can be specified when creating multiple workers with celery multi:

celery -A proj multi start 2 -P gevent -c 1000

This starts 2 gevent workers, but how can I can specify pool implementation on a per worker basis when using celery multi, so that one worker uses a gevent pool and the other uses prefork?

The celery multi docs don't mention anything regarding this specific matter, and the source code (celery.bin.multi) didn't really show that this was possible (unless I misread/misunderstood the code).

Answer 1

So currently the only way to specify per worker Pool implementations is by running independent celery worker commands:

$ celery -A proj worker start -P gevent -Q:queue1 -c 500
$ celery -A proj worker start -P prefork -Q:queue2 -c 4

celery multi does not support -P:worker1 gevent, -P:worker2 prefork. This makes things difficult when using the init.d script provided within the Celery docs to daemonize celeryd. So you either have to modify the init.d script or use something like Supervisor.

Answer 2

There is a way to start different implementations using celery multi. Just like Amir R. said about -Q option.

celery multi start 2 -A default_prefork_queue -Q:2 gevent_queue_name -P:2 gevent -c:2 1000

By default it's prefork option selected.

In example above celery creates one worker with gevent pool with 1000 limit and one prefork worker with default limit of processes.

More examples here