Django + Celery with long-term scheduled tasks

Question

I'm developing a Django app which relies heavily on Celery task scheduling, using Redis as backend. Tasks can be set to run at a large periods of time, as well as in a few seconds/minutes.

I've read about Redis visibility timeout and consequences of scheduling tasks with timedelta greater than visibility timeout (I'm also in the process of dealing with it in a previous project), so I'm interested if there's anything neater than my solution, which is to have another "helper" task run 5 minutes before the "main" one needs to be executed, scheduling the "main" task to run in required time, storing task id in DB, and then checking in "main" task if the stored task id is the one that is being run. The last part (with task id storing) is required as multiple runs of "helper" task could spawn a lot of "main" task instances, but with this approach each will have different task id.

I really hate how that approach sounds and how it works, as if the task is scheduled to be run a month from current time, "helper" and "main" tasks are executed up to a hundred times.

I also know that it's an open issue, so I'm interested in more a neat workaround than a solution itself.

Why not use django-celery-beat with cronschedule tasks (pypi.org/project/django_celery_beat)? — 2ps
@2ps, celery beat was created to run periodic tasks and seems to be overcomplicated if used just to solve the problem in question — pomo_mondreganto

pomo_mondreganto pomo_mondreganto · Accepted Answer · 2019-04-07T08:43:32

Having tested available options, in my opinion only using RabbitMQ as broker solves the whole problem.

Although it's a viable option for me, lack of some of redis configuration parameters (e.g. pool size) makes it unusable for those who are using hosting services with some limit on opened broker connection.

Django + Celery with long-term scheduled tasks

1 Answers