I want to use multiprocessing in python(specifically Pool Class) in windows(zeppelin notebook). But after reading a lot I got to know that the Pool Class uses pickle in its back end and can be use for only the objects that can be pickled. As I can't serialize a function or a class using pickle in Python(at least in 3.6 version), I found out that pathos.multiprocessing library can be used to perform multiprocessing as it uses dill instead of pickle in the back end which according to dill package documentation can serialize anything. But when I used it I got errors. Below is the simple code that I used:
import pathos.multiprocessing as mp
def cube(x):
return x**3
p = mp.Pool(4)
p.apply(cube, [1,2,3])
And this is the error I got:
py4j.protocol.Py4JError: An error occurred while calling o0.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
at py4j.Gateway.invoke(Gateway.java:279)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:748)
I got the same error even if I used map, apply_async or map_async. Can someone help me finding out the issue.
Thanks in advance
_pickle.PicklingError: Can't pickle <function cube at 0x7f3283ee3048>: attribute lookup cube on __main__ failed. - Shubham Kedia