Parallel Python: Passing a function written in another module to 'submit'

Question

I am using the Parallel Python module (pp), and want to submit a job to a worker. However, the function that I want to execute is in another module (written with Cython), and I don't know how to import the function name to the new worker. The method suggested here, i.e importing the module "walkerc" inside the function cannot work since walk itself is defined in walkerc, from the filename "walkerc.so"

import pp
from walkerc import walk
# Other stuff here
ser = pp.Server()
# Some more definitions
ser.submit(walk, (it, params))
ser.submit(walk, (1000, params), modules = ("walkerc",), globals = globals())

Both the statements above fail, I get the following error:

Traceback (most recent call last):

File "", line 1, in ser.submit(walk, (1000, params), modules = ("walkerc",), globals = globals())

File "/usr/lib/python2.7/site-packages/pp.py", line 458, in submit sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)

File "/usr/lib/python2.7/site-packages/pp.py", line 629, in __dumpsfunc sources = [self.__get_source(func) for func in funcs]

File "/usr/lib/python2.7/site-packages/pp.py", line 696, in __get_source sourcelines = inspect.getsourcelines(func)[0]

File "/usr/lib/python2.7/inspect.py", line 690, in getsourcelines lines, lnum = findsource(object)

File "/usr/lib/python2.7/inspect.py", line 526, in findsource file = getfile(object)

File "/usr/lib/python2.7/inspect.py", line 420, in getfile 'function, traceback, frame, or code object'.format(object))

TypeError: '<'built-in function walk'>' is not a module, class, method, function, traceback, frame, or code object

The function 'walk' itself is imported properly within the main program, it is the process of submitting it to a new worker that is problematic.

How can I specify the function name 'walk' properly? I do not want to define 'walk' in the same file as which I have called it because I have modified it in Cython and want to have better performance. Is there an alternative?

famousgarkin famousgarkin · Accepted Answer · 2014-06-24T11:08:48

Try renaming your walk function to something else, mywalk for example. As the exception text suggests, your environment seems to have a built-in function that goes by the name walk, so the inspect module gets confused.

I can successfully pass my imported walk function like this on my system, no conflict here and nothing more needed, the function gets executed using the given argument:

import pp
from walkerc import walk

pps = pp.Server()
pps.submit(walk, args=(1,))

But passing dir, which is a built-in function for sure:

pps.submit(dir)

I get the exact same error as you do:

Traceback (most recent call last):
  File "parallel.py", line 9, in 
    pps.submit(dir)
  ...
  File ".../lib/python2.7/inspect.py", line 420, in getfile
    'function, traceback, frame, or code object'.format(object))
TypeError:  is not a module, class, method, function, traceback, frame, or code object

Update after the below discussion:

So the problem here is that Python treats the members that come from C extensions as built-ins. The code above works with the regular Python module, but I was able to replicate the OP's error when importing and passing the function from a C extension.

Therefore I wrapped the C extension function call inside a normal Python function, which does the trick. Note that now the walk function import was moved to the wrapping function, so that it can construct it's own context itself when dispatched.

import pp

def walk(n):
    import walkerc
    return walkerc.walk(n)

def print_callback(result):
    print('callback: ', result)

pps = pp.Server()
job = pps.submit(walk, args=(1,), callback=print_callback)

Parallel Python: Passing a function written in another module to 'submit'

1 Answers

Update after the below discussion: