0
votes

I want to use multiprocessing in python(specifically Pool Class) in windows(zeppelin notebook). But after reading a lot I got to know that the Pool Class uses pickle in its back end and can be use for only the objects that can be pickled. As I can't serialize a function or a class using pickle in Python(at least in 3.6 version), I found out that pathos.multiprocessing library can be used to perform multiprocessing as it uses dill instead of pickle in the back end which according to dill package documentation can serialize anything. But when I used it I got errors. Below is the simple code that I used:

import pathos.multiprocessing as mp

def cube(x):
    return x**3

p = mp.Pool(4)

p.apply(cube, [1,2,3])

And this is the error I got:

py4j.protocol.Py4JError: An error occurred while calling o0.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
    at py4j.Gateway.invoke(Gateway.java:279)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:209)
    at java.lang.Thread.run(Thread.java:748)

I got the same error even if I used map, apply_async or map_async. Can someone help me finding out the issue.

Thanks in advance

1
For the example you gave, the standard multiprocess library will work. Are you trying to map something more complicated? - James
Hi James, You can't use standard multiprocessing library as well for doing this. When you do this using standard library you get the pickle error - _pickle.PicklingError: Can't pickle <function cube at 0x7f3283ee3048>: attribute lookup cube on __main__ failed. - Shubham Kedia
Are you trying to run in interactive mode? - James
Yes I am using a zeppelin notebook - Shubham Kedia
@ShubhamKedia did you solve this problem by any means? - Andrzej Sołtysik

1 Answers

0
votes

Also If i use the code like the this:

import multiprocessing as mp

def cube(x):
    return x**3

if __name__ == '__main__':
    p = mp.Pool(4)
    p.apply(cube, [1,2,3])

I don't see any output returned. Not sure what the issue is.