Showing tqdm progress bar while using Python multiprocessing

Question

I am trying to run some computationally heavy task using Python's multiprocessing library, and I would like to show a tqdm progress bar for each worker. Specifically, I would prefer to have this functionality for multiprocessing.Process workers or multiprocessing.Pool workers.

I am aware of the similar StackOverflow questions about this topic (see e.g. (1) Multiprocessing : use tqdm to display a progress bar, (2) Show the progress of a Python multiprocessing pool imap_unordered call?, (3) tqdm progress bar and multiprocessing ) but they all seem interested on showing one progress bar across all workers. I would like to show a progress bar for each worker.

Here is an example function, taking place of my computationally expensive function I would like to multiprocess:

from tqdm import notebook
import time
def foo2(id):
    total = 100
    with notebook.tqdm(total=total, position=id) as pbar:
        for _ in range(0, total, 5):
            pbar.update(5)
            time.sleep(0.1)

When I try this sequentially, I get the expected results: 5 progress bars filling up one after the other.

However, when I try to do this with multiprocessing, I get the desired speed-up, but no progress bars are displayed. This is true whether I use Pool workers or Process workers. Here is my sample code:

%%time
from multiprocessing import Pool
pool = Pool(5)
pool.map(foo2, range(5))
pool.close()
pool.join()

Pool - no progress bars

Per the comments here (https://github.com/tqdm/tqdm/issues/407#issuecomment-322932800), I tried using several ThreadPool workers, and this strangely was able to produce the progress bars. However, for my situation, I would prefer to use Pool or Process workers with progress bars.

%%time
from multiprocessing.pool import ThreadPool
pool = ThreadPool(5)
pool.map(foo2, range(5))
pool.close()
pool.join()

ThreadPool - progress bars show!

Hopefully someone can help me with this. I have tried just about everything I could think of. For reference, I am using Python 3.7.7 and tqdm 4.57.0.

let me get this straight, you want 1 process for each progress bar and every progress bar should display accordingly with the assigned task? — alexzander
Yes, exactly. I would like to run N processes and display N progress bars, each bar corresponding to its own particular process. — andytaylor823
you need N + 1 processes, N for the progress bars and 1 for the sys.stdout that communicates with other N processes, in order to achieve what you want. that single process should always check for the others to see their progress and then, update to the screen. im not quite sure about that what im saying can be implemented, but you always have to be positive and try. — alexzander
Thank you for your reply. Since I provided short snippets of code, I am still looking for an answer that is able to address my specific situation. I believe the setup of this problem is fairly simple, yet the solution still eludes me. I have run out of possibilities to try, which is why I have posted my question here. — andytaylor823
maybe this will help you: pypi.org/project/tqdm-multiprocess. — alexzander

andytaylor823 andytaylor823 · Accepted Answer · 2021-02-26T19:00:10

Searching the issues posts on the main github page for tqdm, I found a hack that works for me, but it definitely feels like a "hack" instead of a true issue fix: https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308

The new (working) code looks like:

from tqdm import notebook
import time
def foo2(id):
    total = 100
    print(' ', end='', flush=True)
    for _ in notebook.tqdm(range(0, total, 5)):
        time.sleep(0.1)

plus

%%time
pool = Pool(5)
#pool.map(foo2, range(5)) # this also works fine with the new hack
for i in range(5):
    pool.apply_async(foo2, args=(i,))
pool.close()
pool.join()

Showing tqdm progress bar while using Python multiprocessing

1 Answers