Celery multiple workers but only one beat worker

0
votes

Im trying to allow users to schedule a periodic task. Im also running multiple celery workers in a container. My command for that container used to look like this:

celery worker -c 4 -B -l INFO -A my.celery.app.celery --scheduler my.celery.scheduler.SchedulerClass

but what happened was that the scheduled task ran 4 times when the time came to run the task.

so i read that you should have a dedicated worker for beat. I changed my command to this one:

celery worker -c 4 -l INFO -A my.celery.app.celery

and added another container exactly like that one that runs the command:

celery -l INFO -B -A my.celery.app.celery --scheduler my.celery.scheduler.SchedulerClass

hoping that now that there is only one beat, there will be no duplicate tasks. But I still get 4 tasks running instead of one.

Any ideas on how this should be done will be helpful

1
celerycelerybeatperiodic-task

1 Answers

2
votes

From the documentation:

You can also embed beat inside the worker by enabling the workers -B option, this is convenient if you’ll never run more than one worker node, but it’s not commonly used and for that reason isn’t recommended for production use:

$ celery -A proj worker -B

So you're likely required to run the beat independently, using:

celery -l INFO -A my.celery.app.celery beat --scheduler my.celery.scheduler.SchedulerClass