2
votes

I am running python 3.4.3 on Windows Standard Embedded 7. I have a class that inherits multiprocessing.Process.

In the class's run method I create a thread for the process object to start.

While watching Task Manager, specifically the Command Line column, when the process class is instantiated I see a 'from.multiprocessing.spawn import spawn_main(parent_pid=XXXX, pipe_handle=XXXX)"" --multiprocessing-fork'.

When the thread in the process starts I see another pythonw.exe multiprocessing fork from the same parent process id. When the thread finishes the separate process ends.

Why does the creation of thread in a separate process cause another multiprocessing fork to spawn?

Thanks for any insight. Will post if it will help, but figured I would ask more generically if this is expected behavior.

EDIT

Sorry it took a bit to some test code together to demonstrate the behavior I am seeing. Unfortunately I neglected to mention that I was also passing in a multiprocessing.Manager Namespace object to the process object. The code below demonstrates what I thought should happen, multiple threads spawn in the child process and only one multiprocessing fork is displayed in Task Manager.

import multiprocessing
import threading
import time

class Comm(multiprocessing.Process):
    def __init__(self):#, namespace=None):
        multiprocessing.Process.__init__(self)
        #self.namespace=namespace
        self.comm_queue=multiprocessing.Queue()

    def talk(self):
        counter=0
        while counter != 4:
            self.comm_queue.put('i am talking')
            time.sleep(2)
            counter += 1

    def yell(self):
        counter=0
        while counter != 3:
            self.comm_queue.put('I AM YELLING')
            time.sleep(5)
            counter += 1

    def make_threads(self):
        self.talk_thread=threading.Thread(target=self.talk)
        self.yell_thread=threading.Thread(target=self.yell)


    def run(self):
        self.make_threads()
        self.talk_thread.start()
        self.yell_thread.start()
        while True:
            time.sleep(1)


if __name__=='__main__':
    #test_manager=multiprocessing.Manager()
    #test_ns=test_manager.Namespace()
    test=Comm()#namespace=test_ns)
    test.start()
    while True:
        message=test.comm_queue.get()
        print(message)

However, if you uncomment everything and pass in the Namespace object, I see two multiprocessing forks spawn. Why does this happen with the multiprocessing.Manager() / Namespace() is included with the process object?

2
Can you share some sample code that reproduces this? - dano
Added sample code, forget to mention that I was using a multiprocessing.Manager and Namespace. If it don't include those then the program runs as I expected. But if I include a Manager/Namespace I see multiple multiprocessing forks in Windows Task Manager. - user2662241

2 Answers

2
votes

multiprocessing.Manager works by spawning a separate Manager server process, which will run until the Manager is garbage collected:

Managers provide a way to create data which can be shared between different processes. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

So, the two processes you see are expected; one is your multiprocessing.Process subclass, and the other is the multiprocessing.Manager server process.

0
votes

This is an easily overlook-able point.

To elaborate the implication for future readers:

If a complex app makes use of multiple multiprocess Queues, dicts and or lists, and each time acquires them via call to a new multiprocess.Manager() object, it would end up in "too many" processes that show on the OS!

It did in mine; and thanks to Dano's input in this thread, the issue got resolved!

While acquiring dicts from a shared multiprocess.Manager() however, one noteworthy issue to beware of, as of Py 361, is: https://bugs.python.org/issue30256