Parallel processing in Python when clearing all variables

Question

I'm trying to run parallel process using python. I managed to do that using the multiprocessing module.

The problem is that when I add a "Clear all variables" at the beginning of the code, the algorithm gives me this error:

Traceback (most recent call last):

File "", line 1, in runfile('C:/Users/cuca/Dropbox/USP/Pesquisa/PosQuali/Paralelo/Paraleltest2.py', wdir='C:/Users/cuca/Dropbox/USP/Pesquisa/PosQuali/Paralelo')

File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile execfile(filename, namespace)

File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile exec(compile(scripttext, filename, 'exec'), glob, loc)

File "C:/Users/cuca/Dropbox/USP/Pesquisa/PosQuali/Paralelo/Paraleltest2.py", line 36, in answer1 = result1.get(timeout=1)

File "C:\Python27\lib\multiprocessing\pool.py", line 563, in get raise TimeoutError

TimeoutError

When I delete the "Clear All Variables" procedure from the beginning of the algorithm, it works.

Any idea why?

Thanks.

#Clear All
all = [var for var in globals() if var[0] != "_"]
for var in all:
    del globals()[var]


from multiprocessing import Pool
import math


def Func1(A):
    for i in range(1000,100000):
        R = A * math.cos(i) * math.cos(i)
    return R


if __name__ == '__main__':

    pool = Pool()
    result1 = pool.apply_async(Func1, [2])   
    result2 = pool.apply_async(Func1, [4])    
    answer1 = result1.get(timeout=1)
    answer2 = result2.get(timeout=1)

Don Kirkby Don Kirkby · Accepted Answer · 2016-06-09T18:59:05

It probably has to do with the restrictions of using multiprocessing on Windows.

Since Windows lacks os.fork() it has a few extra restrictions.

It may be that some global variables are used to control multiprocessing under Windows, and you're breaking that. Try printing the variable names before you delete them.

You'll get more helpful answers if you describe why you want to clear out all the global variables. I would guess that you'd be better off to use fewer global variables, so you don't have to clear them out.

Parallel processing in Python when clearing all variables

1 Answers