Getting same thread id inside celery workers

Question

I tried getting the current thread id and process id while running a task inside a celery worker process. I have set prefetch multiplier to 1 and I have 4 cpu core machine, so there will be 4 worker processes running for each workers. I have only 1 worker running ( 4 worker processes).

As per my understanding each of the worker processes actually handles the execution of task. When I run 4 tasks simultaneously I tried getting the process id and thread id inside the task using os.getpid() and threading.get_ident() respectively.

To no surprise, for every task running I got the same set of 4 process_id (as there are 4 worker processes running), but the thread id for each of the process are same. I am not able to understand how is this possible.

Following is my observations when running tasks:

>     log: pid id: 513, t_id 140373758563328
>     log: pid id: 514, t_id 140373758563328
>     log: pid id: 513, t_id 140373758563328
>     log: pid id: 513, t_id 140373758563328
>     log: pid id: 513, t_id 140373758563328
>     log: pid id: 513, t_id 140373758563328
>     log: pid id: 578, t_id 140280371217408
>     log: pid id: 579, t_id 140280371217408

Answer 1

It is pretty much possible that the same worker process picks several successive tasks, especially if it executes tasks faster than you are sending. If you prefer a more random behaviour, pass the -Ofair optimisation parameter to your workers (something like: celery -A my.project.app worker -c 5 -O fair -l info)

Answer 2

The thread if fro the Threading library is probably not the correct way to get a Celery worker's thread id. That's because Celery uses different libraries for concurrency and it depends on your configuration. By default these pools of workers are managed by billiard (also a Celery project library) and the correct way to get the process info would be to use current_process