How can I print from a Python multiprocessing process?

Question

Edit: For those finding this later, this is explicitly not a duplicate of the other topic that somebody linked to this. The problem is unrelated to stdout buffering, but rather was a misunderstanding of how imap_unordered was calling do_something

I'm trying to debug a separate issue which requires some print statements in my Python multiprocessing code. I can print() fine in the main processes, but when I spawn new processes, I can't successfully print anything.

Here is a bare bones example of my issue:

import argparse
from multiprocessing import Pool, get_context


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--foo', required=True)
    global args # need this if I'm reading from threads / forks
    args = parser.parse_args()

    print('This prints fine')
    with get_context('spawn').Pool(processes=4) as pool:
        pool.imap_unordered(do_something, [0, 1, 2, 3, 4 ])

    return

def do_something():
    print(args.foo * 2)
    print("this doesn't print either")

if __name__=="__main__":
    main()

What is the correct way to print from the do_something method?

Please note that I am using python from Bash on an Ubuntu 18.04 machine. I'm not using IDLE, or any IDE, which I see similar questions for.

Edit: Please note that the issue is also not that the printing is delayed, but rather that it does not happen at all. Even if I flush the stdout buffer in the do_something function.

Final edit: It looks like my code was not actually calling the do_something function. When I force it to through the methods described in below, I do see the expected prints. This code produces that output:

import collections
from multiprocessing import Pool, get_context


def main():
    print('This prints fine')
    with get_context('spawn').Pool(processes=4) as pool:
        results = collections.deque(pool.imap_unordered(do_something, [0, 1, 2, 3, 4 ]), 0)

    print(results)
    return

def do_something(i):
    print("this doesn't print either") # Actually does print
    return i


if __name__=="__main__":
    main()

Why are you trying to call imap_unordered with a function that takes no arguments? — user2357112 supports Monica
@user2357112 supports Monica Just a typo. This is based on code that does something different which I'm trying to debug. I'll add an argument to avoid confusion. Thanks. — A. P.
Also, your workers don't have an args global. Despite how hard multiprocessing tries to pretend to be threading, it's nothing like threading, and one of the ways that manifests is that processes don't share variables. — user2357112 supports Monica
@user2357112 supports Monica that's actually what I'm trying to debug (trying to initialize the spawned processes with the args of the parent), but I thought it was too complicated to put in a question. Wanted to take it one step at a time. Perhaps I'll post this differently with the larger problem. — A. P.
@JohnGordon unfortunately that does not answer. Even if I import sys and flush stdout all within that do_something method, still nothing prints. — A. P.

Weeble Weeble · Accepted Answer · 2020-07-21T00:01:53

I said that imap_unordered is lazy and while that's true, it's not actually the reason your code doesn't run. Your code doesn't run because nothing causes the main process to wait for the pool children to finish before terminating the pool. The pool is terminated when the with block exits. A straightforward way to wait is to iterate through the results of imap_unordered.

~~imap_unordered is lazy. So lazy, in fact, that it isn't actually running your code yet. You'll need to iterate it to retrieve its return values.~~ That's not all that's wrong with your program, but that should at least get you moving forward!

How can I print from a Python multiprocessing process?

2 Answers