Specify Worker in Celery

Question

I have two workers:

celery worker -l info --concurrency=2 -A o_broker -n main_worker
celery worker -l info --concurrency=2 -A o_broker -n second_worker

I am using flower to monitor and receive API requests for these workers:

flower -A o_broker

to launch these celery workers from an API I use flower per the docs:

curl -X POST -d '{"args":[1,2]}' 'http://localhost:5555/api/task/async-apply/o_broker.add'

However, with this POST request it runs the task on either one of the workers. I need to choose to run a specific broker to complete the task.

How do I specify or set this up so I can choose what worker to use for the add task? If you have a solution using another API without flower, that would also work.

Answer 1

The easiest way to achieve this is with separate queues. Start worker with -Q first_worker,celery and the second broker with -Q second_worker,celery. celery is the default queue name in celery.

Now, when you want to send a task to just the first worker, you can route the task to the first_worker queue using celery's task_routes setting. You treat routing tasks to the second_worker queue symmetrically. You can also manually route a particular task call to a certain queue when using apply_async, e.g.:

add.apply_async(args=(1, 2), queue='first_worker')

n.b., last I checked, flower will only monitor one of your queues (by default it's the celery queue).