qt - Multiprocessing working in Python but not in iPython

14
votes

I am running the following code in iPython:

import multiprocessing

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x


if __name__ == '__main__':
    pool = multiprocessing.Pool()
    results = pool.map(my_function, [1,2,3,4,5,6])
    print(results)

in ipython QT console on Windows. However, the code does not work -- the QT console just freezes up. The issue is specific to iPython (the code above should work for the regular Python 2.7).

Any solution to this?

2
pythonqtmultiprocessingipython
Works fine on Ubuntu. Must be an issue with windows. Do you paste the code into Ipython? - Padraic Cunningham
Yes, I do paste the code. - user3438258
does it work when you run it from another shell or IDE? - Padraic Cunningham
Yes, I just tried it in Spyder IDLE. It works there. Any idea how to make it work in the actual QT console? - user3438258
I second @PadraicCunningham's comment. Works fine on Linux distributions (I'm using Debian). - lostsoul29

2 Answers

13
votes

From the documentation:

Note

Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter

7
votes

At least in the current version of Jupyter Notebook (the successor of IPython) you can solve this by moving the target function to a separate module, and importing it.

I have no idea why this works, it's rather odd, but it does.

i.e. - in a workers.py file put

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x

Then in IPython/Jupyter notebook put:

import multiprocessing
import workers

pool = multiprocessing.Pool()
results = pool.map(workers.my_function, [1,2,3,4,5,6])
print(results)

also - the if-main thingy doesn't seem to be needed.

Credit: Gaurav Singhal