Celery all generated tasks status

Question

Django produces multiple Celery tasks through chains in one script run (f.e. if / is opened in browser, 1000 tasks are called by delay method).

I need something that will restrict new task generation, if tasks, queued in previous script run, are still running.

Answer 1

You need a distributed lock for this, which celery doesn't offer natively.

For these kinds of locks I've found redis.Lock useful to most cases. If you need a semaphore, you can use redis' atomic incr/decr functions along with some kind of watchdog mechanism to ensure your processes are still running.

Answer 2

You can restrict the number of tasks of one type running at the same time by setting:

rate_limit = “1000/m” 

=> only 1000 tasks of this type can run per minute. (see http://docs.celeryproject.org/en/latest/userguide/tasks.html#list-of-options)