Celery multi workers unexpected task execution order

Question

I run celery:

celery multi start --app=myapp fast_worker 
       slow_worker
       -Q:fast_worker fast-queue      
       -Q:slow_worker slow-queue
       -c:fast_worker 1 -c:slow_worker 1
       --logfile=%n.log --pidfile=%n.pid

And celerybeat:

celery beat -A myapp

Task:

@task.periodic_task(run_every=timedelta(seconds=5), ignore_result=True)
def test_log_task_queue():
    import time
    time.sleep(10)
    print "test_log_task_queue"

Routing:

CELERY_ROUTES = {
    'myapp.tasks.test_log_task_queue': {
        'queue': 'slow-queue',
        'routing_key': 'slow-queue',
    },
}

I use rabbitMQ. When I open rabbitMQ admin panel, I see that my tasks are in slow-queue, but when I open logs I see task output for both workers. Why do both workers execute my tasks, even when task not in worker queue?

Nikitka Nikitka · Accepted Answer · 2017-05-04T11:01:29

It looks like celery multi creates something like shared queues. To fix this problem, I added -X option:

celery multi start --app=myapp fast_worker 
   slow_worker
   -Q:fast_worker fast-queue      
   -Q:slow_worker slow-queue
   -X:fast_worker slow-queue
   -X:slow_worker fast-queue
   -c:fast_worker 1 -c:slow_worker 1
   --logfile=%n.log --pidfile=%n.pid

Celery multi workers unexpected task execution order

1 Answers