Celery, RabbitMQ and Django on Heroku: Reaching Memory Limit

Question

I'm reaching the RAM memory limit on my Heroku Dyno (hobby) currently while running Celery tasks with RabbitMQ in a Django app.

I'm played around with the Celery settings a little bit but I keep hitting the memory limit, and I'm missing the technical knowledge on memory optimization. I'm wondering if there is anything I can do with my current settings to prevent reaching the limit or is the only solution here to upgrade the Heroku Dyno?

BROKER_URL = 'amqp://url'
BROKER_POOL_LIMIT = 5
CELERY_RESULT_BACKEND = None
CELERY_MAX_TASKS_PER_CHILD = 10
CELERY_MAX_MEMORY_PER_CHILD = 80000

Procfile:web: gunicorn app_name.wsgi worker: celery -A app_name worker -l info --without-heartbeat

Task.py file:@shared_task(acks_late=True, ignore_result=True)
def function_name(args):

JDenais JDenais · Accepted Answer · 2020-08-13T13:28:02

I had the same issue and setting the max concurrency to 2 or 3 solved the problem.

In your procfile, add -c 2 or -c 3 in the worker definition.

By default, Celery will set the concurrency to the number of CPUs available. And the concurrency defines the number of celery workers that will spawn.

Celery, RabbitMQ and Django on Heroku: Reaching Memory Limit

1 Answers