Does eventlet with celery executes tasks in parallel?

Question

This post is in continuation with my previous post - celery how to implement single queue with multiple workers executing in parallel?

I implemented celery to work with eventlet using this command :-

celery -A project worker -P eventlet -l info --concurrency=4

I can see that my tasks are getting moved to active list faster (In flower) but i am not sure if they are executing in parallel? I have a 4 core server for production but I am not utilizing all the cores at the same time.

My question is :-

how can I use all 4 cores to execute tasks in parallel?

Answer 1

Both eventlet/gevent worker types provide great solution for concurrency at the cost of stalling parallelism to 1. To have true parallel task execution and utilise cores, run several Celery instances on same machine.

I know this goes counter to what popular Linux distros have in mind, so just ignore system packages and roll your great configuration from scratch. Systemd service template is your friend.

Another option is to run Celery with prefork pool, you get parallelism at the cost of stalling concurrency to number of workers.