Celery active tasks persistence

Question

If celery crashes during executing some task, this task is lost after restart celery. Tasks that in the queue in the moment of the crash, will be restored fine in RabbitMQ. But how can I make active tasks persistent?

Answer 1

Celery is configured by default with task_acks_late=False. [1] This means that the task is acked as soon as the worker receives it from the queue. And if the task fails, the queue has no way of knowing it.

Set task_acks_late to True and the task will be acked after it has been processed. When the task fails, it is requeued. [2] But be careful, your tasks must be idempotent. [3]

• [1] http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-task_acks_late
• [2] http://docs.celeryproject.org/en/latest/faq.html#faq-acks-late-vs-retry
• [3] https://en.wikipedia.org/wiki/Idempotence