django/celery multiple queues not consuming any task

Question

I'm new to celery and django In my celery setup when I invoke tasks without any queues it working perfectly with multiple workers. But When I specify queues the workers don't consume anything

I have a project called example here the structure

├── example
│   ├── celery.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── mailer
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   ├── __init__.py
│   │   └── __pycache__
│   │       └── __init__.cpython-36.pyc
│   ├── models.py
│   ├── tasks.py
│   ├── tests.py
│   └── views.py
└── manage.py

In settings.py (Queue setting)

# ==============================
# CELERY SETTINGS
# ==============================
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Kolkata'
CELERY_ROUTES = {
    'example.mailer.tasks.first_task': {'queue': 'first_queue'},
    'example.mailer.tasks.second_task': {'queue': 'second_queue'},
}

And my mailer/tasks.py

@shared_task
def first_task():
    for i in tqdm(range(100000000)):
        pass
    return "First task finished"


@shared_task
def second_task():
    for i in range(100000000):
        print(i,'Second Task')
    return "Second task finished"

with this when I run workers like

celery -A example worker -l info -c 4 -n worker1
celery -A example worker -l info -c 2 -n worker2

It's working perfect but when I try

celery -A example worker -l info -c 2 -n worker1 -Q first_queue
celery -A example worker -l info -c 2 -n worker2 -Q second_queue

It's not working.Here some screen shots

first_queue worker running
second_queue worker running
What happens when I call them
But tasks are not performing

I hope someone can help me. Thanks in advance

Stuart Dines Stuart Dines · Accepted Answer · 2018-09-17T07:06:26

Based upon the directory structure you have shown mailer is its own application and should be declared without example at the start e.g.

CELERY_ROUTES = {
    'mailer.tasks.first_task': {'queue': 'first_queue'},
    'mailer.tasks.second_task': {'queue': 'second_queue'},
}

django/celery multiple queues not consuming any task

1 Answers