Assign different tasks to different celery workers

Question

I am running my server using this command:

celery worker -Q q1,q2 -c 2

which shows that my server will handle all the tasks on queues q1 and q2, and I have 2 workers running. My server should support 2 different tasks:

@celery.task(name='test1')
def test1():
    print "test1"
    time.sleep(3)

@celery.task(name='test2')
def test2():
    print "test2"

If I send my test1 tasks to queue q1 and test2 to q2, both workers will run both tasks. So the result will be:

test1
test2
test1
test2
...

Now what I need is one of my workers handle test1 and the other one handles test2. One solution is to run two celery workers like this:

celery worker -Q q1 -c 1
celery worker -Q q2 -c 1

And each one handles 1 queue. But I would like to have them cleaner and use -c 2. I found Celery Routing but am not sure if that is what I want.

AliBZ AliBZ · Accepted Answer · 2013-10-29T23:04:49

I found the answer and I am putting it here in case someone else wanted to do the same:

Instead of using celery worker -Q q1,q2 -c 2, celery multi could be used:

celery multi start 2 -Q:1 q1 -Q:2 q2 -c:1 1 -c:2 1

Which says that we have 2 queues: -Q:1 q1 means queue #1 with name of q1 and same for q2 and we have different concurrencies for each queue, -c:1 1 means the first celery worker has a concurrency of 1.

Assign different tasks to different celery workers

1 Answers