python - yet another confusion with multiprocessing error, 'module' object has no attribute 'f'

69
votes

I know this has been answered before, but it seems that executing the script directly "python filename.py" does not work. I have Python 2.6.2 on SuSE Linux.

Code:

#!/usr/bin/python
# -*- coding: utf-8 -*-
from multiprocessing import Pool
p = Pool(1)
def f(x):
    return x*x
p.map(f, [1, 2, 3])

Command line:

> python example.py
Process PoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap
    self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 339, in get
    return recv()
AttributeError: 'module' object has no attribute 'f'
5
pythonmultiprocessing
possible duplicate of Using python multiprocessing Pool in the terminal and in code moudles for Django or Flask – jb.
@jb. that post is much later than this one, this was 2010, that one is 2013 – gatoatigrado
Age is irrevelant there is consensus on meta that question with better answer should be chosen, and another one has IMO better answer. – jb.

5 Answers

133
votes

Restructure your code so that the f() function is defined before you create instance of Pool. Otherwise the worker cannot see your function.

#!/usr/bin/python
# -*- coding: utf-8 -*-

from multiprocessing import Pool

def f(x):
    return x*x

p = Pool(1)
p.map(f, [1, 2, 3])
6
votes

This one works:

#!/usr/bin/python
# -*- coding: utf-8 -*-
from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == "__main__":
    p = Pool(1)
    p.map(f, [1, 2, 3])

I'm not 100% sure why your code does not work, but I guess the reason is that child processes launched by the multiprocessing module try to import the main module (to have access to the methods you defined), and the if __name__ == "__main__" stanza is required not to execute the initialization code where you set up your pool.

1
votes

One possibility is that your python file has the same name as a module:

  • test.py
  • test/
    • __init__.py

in pickle.py, you have the error coming from:

    def find_class(self, module, name):
      # Subclasses may override this
      __import__(module)
      mod = sys.modules[module] # <- here mod will reference your test/__init__.py
      klass = getattr(mod, name)
      return klass
1
votes

The problem I had was solved by using if __name__ == "__main__" as pointed out by Tamás; in Eclipse for Windows the examples do not work under the interpreter. This is explained in http://docs.python.org/2/library/multiprocessing

1
votes

This comes from the fact that with p = Pool(1) the main process forks processes (threads vs processes) before it creates the function f. As stated in Bartosz answer the spawned processes do not have access to the new function.

def f1(x):
    ...

p = Pool(1) # p is spawned and is now an independent process, knows f1

def f(x): # p doesn't not share this object
    ...