0
votes

Let's start from the punch line:

✗ ~/.pyenv/versions/3.7.11/bin/python demo_multiprocessing.py
<Process(Process-6, started)>  is putting  4  in the list
<Process(Process-2, started)>  is putting  0  in the list
<Process(Process-4, started)>  is putting  2  in the list
<Process(Process-5, started)>  is putting  3  in the list
<Process(Process-3, started)>  is putting  1  in the list
<Process(Process-7, started)>  is putting  5  in the list
all processes finished, here's what we have in the list:
[4, 0, 2, 3, 1, 5]
✗ ~/.pyenv/versions/3.9.11/bin/python demo_multiprocessing.py
<Process name='Process-4' parent=74535 started>  is putting  2  in the list
<Process name='Process-5' parent=74535 started>  is putting  3  in the list
<Process name='Process-6' parent=74535 started>  is putting  4  in the list
<Process name='Process-2' parent=74535 started>  is putting  0  in the list
<Process name='Process-7' parent=74535 started>  is putting  5  in the list
<Process name='Process-3' parent=74535 started>  is putting  1  in the list
all processes finished, here's what we have in the list:
[]

here's the code:

from multiprocessing import Process, current_process, Manager
from typing import List
from random import shuffle

class StreamInterface:
    def write(self, x: int) -> None:
        raise NotImplemented

class ListStream(StreamInterface):
    def __init__(self, manager: Manager):
        self._manager = manager
        self.shared_list = self._manager.list()

    def write(self, x : int):
        self.shared_list.append(x)

    def pop_all(self) -> List[int]:
        tmp = list(self.shared_list)
        self.shared_list[:] = []
        return tmp


class NullStream(StreamInterface):
    def write(self, x:int):
        pass


n_processors = 6

def do_work(i):
    print(current_process(), " is putting ", i, " in the list")
    ShiftPlanner.list_stream.write(i)

class ShiftPlanner:
    list_stream: StreamInterface = NullStream()
    pass


if __name__ == "__main__":
    with Manager() as manager:
        x = ListStream(manager)
        ShiftPlanner.list_stream = x
        processes = [
            Process(
                target=do_work,
                args=(i,)
            )
            for i in range(n_processors)
        ]

        shuffle(processes)

        for p in processes:
            p.start()

        for p in processes:
            p.join()

        print("all processes finished, here's what we have in the list:")
        print(x.pop_all())
        ShiftPlanner.list_stream = NullStream()

what's going on? Hint: it seems the way class variables are handled changed. If I put a breakpoint at the "write" method, I see that the subprocesses think that self.list_stream is the default, NullStream(), value. But I'd appreciate more details on what changed and what's the most elegant, idiomatic way to fix this - thanks!