multiprocessing.Pool cannot return OrderedDict subclass with additional argument

Question

I am trying to have a simple subclass of OrderedDict that gets created by a Pool then returned.

It seems that the pickling process when returning the created object to the pool tries to re-instantiate the object and fails due to the required additional argument in the __init__ function.

This is a minimal (non) working example:

from collections import OrderedDict
from multiprocessing import Pool


class Obj1(OrderedDict):
    def __init__(self, x, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.x = x


def task(x):
    obj1 = Obj1(x)
    return obj1


if __name__ == '__main__':
    with Pool(1) as pool:
        for x in pool.imap_unordered(task, (1,2,3)):
            print(x.x)

If I do this I get the following error.

Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.6/multiprocessing/pool.py", line 463, in _handle_results task = get() File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv return _ForkingPickler.loads(buf.getbuffer()) TypeError: init() missing 1 required positional argument: 'x'

Again this fails when the task functions returns to the pool and I guess the object gets pickled?

If I changed OrderedDict by a simple dict it works flawlessly....

I have a workaround to use kwargs and retrieve the attribute of interest but I am stumped about the error to start with. Any ideas?

Roland Smith Roland Smith · Accepted Answer · 2019-08-10T08:45:23

You can define __getstate__() and __setstate__() methods for your class.

In those functions you can make sure that x is handled as well. For example:

def __getstate__(self):
    return self.x, self.items()

def __setstate__(self, state):
    self.x = state[0]
    self.update(state[1])

BTW, from CPython 3.6 there is no reason to use OrderedDict, since dictionary order is insertion order. This was originally an implementation detail in CPython. In Python 3.7 it was made part of the language.

multiprocessing.Pool cannot return OrderedDict subclass with additional argument

1 Answers