New RabbitMQ Queue being created for every celery task

Question

I am using celery 3.1 with rabbitmq as broker.
Issue is when I publish a task (i.e task.delay()) message is getting created in rabbitmq with reply-to header containg uniqueid (say 123). Due to this rabbitmq is creating a queue with name as 123 and this is happening for every task that is published.

Say if I run a task for 10 times, 10 reply queues are getting created. I don't even want reply to go to rabbitmq and currently I configured CELERY_RESULT_BACKEND as rpc.Here is the configuration

BROKER_URL = 'amqp://guest@localhost//' CELERY_RESULT_BACKEND = 'rpc://'

Task @app.task(name="tasks.multiplynumbers",bind=True,default_retry_delay=5,max_retries=2,serializer='json',acks_late=True,track_started=True)

def multiply(self,x, y):

try:

    return x * y

except Exception as exc:

    raise self.retry(exc=exc)

Here is the message :

reply_to: 40872e4f-10fc-313d-a48d-13aa94419f20

correlation_id: 25d6f28a-9b18-49c0-950c-19af7af52770

priority: 0

delivery_mode: 2

headers: content_encoding: utf-8

content_type: application/json

Payload

265 bytes Encoding: string {"taskset": null, "task": "reotasks.multiplynumbers", "errbacks": null, "timelimit": [null, null], "callbacks": null, "utc": true, "chord": null, "expires": null, "args": [3, 8], "retries": 0, "id": "25d6f28a-9b18-49c0-950c-19af7af52770", "eta"

when the above message is published then a queue with name 40872e4f-10fc-313d-a48d-13aa94419f20 is getting created.

Did anyone encounter this situation? Any help is greatly appreciated

Prathap reddy Prathap reddy · Accepted Answer · 2016-04-11T21:18:25

After further research, I figured out when celery_result_backend is set to 'rpc://' and use rabbitmq as broker, it seems that every task is published with reply-to header and a new queue gets created based on the information in below link:

https://www.rabbitmq.com/direct-reply-to.html

It would be good if we can have some control on to which queue to publish the results and any best practices to deal with

New RabbitMQ Queue being created for every celery task

1 Answers