How celery chord tasks select queue?

Question

Got a question regarding celery chord queue selection.

I use celery 4.1, python 2.7.10 and rabbitmq 3.5.4 as broker.

What my purpose at the beginning is getting how many chord tasks are pending in the queue.

So I divide the chain task and chord callback task into different queues and may get the pending task number from callback queue depth.

Here is code snippet:

@shared_task(name="analyze_atom", queue="atom")
def analyze_atom(image_urls, targetdir=target_path, studentuid=None):
    return {}


@shared_task(name="summary_up", queue="summary")
def summary_up(rets, studentuid, images):
    return {}


chord(analyze_atom.s([image]) for image in images)(summary_up.s(studentuid, images))

So, analyze_atom is scheduled to queue "atom" while task summary_up to queue "summary".

Now here comes the problem, when I call chord with two elements within images, the expected result is two analyze_atom tasks in "atom" and one summary_up task in "summary", eh?

I use rabbitmq management to inspect queues, however, I only find two tasks in atom and nothing in summary queue, and, "summary" queue is always 0 even when the whole chord tasks are successfully done.

I don't know where the summary_up task is scheduled to?

Anyone know how chord tasks select queues?

Thanks.

Wesley

Jimmy Lee Jones Jimmy Lee Jones · Accepted Answer · 2018-03-26T11:43:48

I've encountered queue issues before, try changing your code to

chord(analyze_atom.s([image]).set(queue="atom") for image in images)(summary_up.s(studentuid, images).set(queue="summary"))

How celery chord tasks select queue?

1 Answers