Let's start from the punch line:
✗ ~/.pyenv/versions/3.7.11/bin/python demo_multiprocessing.py
<Process(Process-6, started)> is putting 4 in the list
<Process(Process-2, started)> is putting 0 in the list
<Process(Process-4, started)> is putting 2 in the list
<Process(Process-5, started)> is putting 3 in the list
<Process(Process-3, started)> is putting 1 in the list
<Process(Process-7, started)> is putting 5 in the list
all processes finished, here's what we have in the list:
[4, 0, 2, 3, 1, 5]
✗ ~/.pyenv/versions/3.9.11/bin/python demo_multiprocessing.py
<Process name='Process-4' parent=74535 started> is putting 2 in the list
<Process name='Process-5' parent=74535 started> is putting 3 in the list
<Process name='Process-6' parent=74535 started> is putting 4 in the list
<Process name='Process-2' parent=74535 started> is putting 0 in the list
<Process name='Process-7' parent=74535 started> is putting 5 in the list
<Process name='Process-3' parent=74535 started> is putting 1 in the list
all processes finished, here's what we have in the list:
[]
here's the code:
from multiprocessing import Process, current_process, Manager
from typing import List
from random import shuffle
class StreamInterface:
def write(self, x: int) -> None:
raise NotImplemented
class ListStream(StreamInterface):
def __init__(self, manager: Manager):
self._manager = manager
self.shared_list = self._manager.list()
def write(self, x : int):
self.shared_list.append(x)
def pop_all(self) -> List[int]:
tmp = list(self.shared_list)
self.shared_list[:] = []
return tmp
class NullStream(StreamInterface):
def write(self, x:int):
pass
n_processors = 6
def do_work(i):
print(current_process(), " is putting ", i, " in the list")
ShiftPlanner.list_stream.write(i)
class ShiftPlanner:
list_stream: StreamInterface = NullStream()
pass
if __name__ == "__main__":
with Manager() as manager:
x = ListStream(manager)
ShiftPlanner.list_stream = x
processes = [
Process(
target=do_work,
args=(i,)
)
for i in range(n_processors)
]
shuffle(processes)
for p in processes:
p.start()
for p in processes:
p.join()
print("all processes finished, here's what we have in the list:")
print(x.pop_all())
ShiftPlanner.list_stream = NullStream()
what's going on? Hint: it seems the way class variables are handled changed. If I put a breakpoint at the "write" method, I see that the subprocesses think that self.list_stream is the default, NullStream(), value. But I'd appreciate more details on what changed and what's the most elegant, idiomatic way to fix this - thanks!