Python Django Celery AsyncResult Memory Leak

Question

The problem is a very serious memory leak until the server crashes (or you could recover by killing the celery worker service, which releases all the RAM used)

There seems to be a bunch of reported bugs on this matter, but very little attention is paid to this warning, In the celery API docs, here

Warning: Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call get() or forget() on EVERY AsyncResult instance returned after calling a task.

And it is reasonable to assume that the leak is related to this warning.

But the conceptual problem is, based on my understanding of celery, that AsyncResult instances are created across multiple Django views within a user session: some are created as you initiate/spawn new tasks in one view, and some you may create later manually (using task_id saved in the user session) to check on the progress (state) of those tasks in another view.

Therefore, AsynResult objects will eventually go out of scope across multiple Views in a real world Django application, and you don't want to call get() in ANY of these views, because you don't want to slow down the Django (or the apache2) daemon process.

Is the solution to never let AsyncResult Objects go out of scope before calling their get() method?

CELERY_RESULT_BACKEND = 'django-db' #backend is a mysql DB

BROKER_URL = 'pyamqp://localhost' #rabbitMQ

Mehran Mehran · Accepted Answer · 2019-02-25T15:51:30

We also faced multiple issues with celery in production, and also tackled a memory leak issue. I'm not sure if our problem scope is the same, but if you don't mind you could try out our solution.

You see we had multiple tasks running on a couple of workers managed by supervisor (all workers were on the same Queue). Now, what we saw that when there were a lot of tasks being queued, the broker (in our case rabbitmq) was sending the amount of tasks our celery workers could process and keeping the rest in memory. This resulted in our memory overflowing and the broker started paginating in our hard drive. We found out from reading the docs that if we allow our broker to not wait for worker results, this issue could be resolved. Thus, in our tasks we used the option,

@task(time_limit=10, ignore_result=True)
def ggwp():
    # do sth

Here, the time limit would close the task after a certain amount of time, and the ignore_result option would allow the broker to just send the task in celery workers as soon as a worker is freed.

Python Django Celery AsyncResult Memory Leak

1 Answers