namespace on python pickle

Question

I got an error when I use pickle with unittest.

I wrote 3 program files:

for a class to be pickled,
for a class which use class in #1,
unittest for testing class in #2.

and the real codes are as follows respectively.

#1. ClassToPickle.py

import pickle
class ClassToPickle(object):
    def __init__(self, x):
        self.x = x
if __name__=="__main__":
    p = ClassToPickle(10)
    pickle.dump(p, open('10.pickle', 'w'))

#2. SomeClass.py

from ClassToPickle import ClassToPickle
import pickle

class SomeClass(object):
    def __init__(self):
        self.pickle = pickle.load(open("10.pickle", 'r'))
        self.x = self.pickle.x
        print self.x

if __name__ == "__main__":
    SomeClass()

#3. SomeClassTest.py

import unittest
from SomeClass import SomeClass
from ClassToPickle import ClassToPickle # REQUIRED_LINE

class SomeClassTest(unittest.TestCase):
    def testA(self):
        sc = SomeClass()
        self.assertEqual(sc.x, 10)

def main():
    unittest.main()

if __name__ == "__main__":
    main()

I ran #1 program first to make pickle file.
And then, when I run program file #2 alone (i.e. enter "python SomeClass.py"), it works.
And, when I run program #3 alone (i.e. enter "python SomeClassTest.py"), it also works.

However, when I run program #3 as "unit-test" in eclipse+pydev, it returns an error message below.

======================================================================
ERROR: testA (SomeClassTest.SomeClassTest)
----------------------------------------------------------------------
Traceback (most recent call last):
$ File "/home/tmp/pickle_problem/SomeClassTest.py", line 9, in testA
sc = SomeClass()
$ File "/home/tmp/pickle_problem/SomeClass.py", line 8, in init
self.pickle = pickle.load(open("10.pickle", 'r'))
$ File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
$ File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
$ File "/usr/lib/python2.7/pickle.py", line 1126, in find_class
klass = getattr(mod, name)
$ AttributeError: 'module' object has no attribute 'ClassToPickle'

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

And also, when I commented out a line that import ClassToPickle class (line 3 in program #3 and commented as "REQUIRED_LINE"), It doesn't work and return an error message described below.

E
======================================================================
ERROR: testA (main.SomeClassTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "SomeClassTest.py", line 9, in testA
sc = SomeClass()
File "/home/tmp/pickle_problem/SomeClass.py", line 8, in init
self.pickle = pickle.load(open("10.pickle", 'r'))
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/usr/lib/python2.7/pickle.py", line 1126, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'ClassToPickle'

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

I guess the problem is about namespace in python, but I don't know what happened exactly and what can I do for resolving it.

How can I "run as unit-test (in eclipse+pydev)" #3 program correctly,
and run #3 program in command line without the line which imports ClassToPickle?
Please help me.

Is there another ClassToPickle.py file anywhere that it could be finding instead of the file you've shown here? — Thomas K
@ThomasK Thanks for your comment, but the answer is No. Only three .py files shown here are in the directory, and I've never create file named ClassToPickle.py. — fbessho

mouad mouad · Accepted Answer · 2011-10-28T12:00:41

That because __main__.ClassToPickle != ClassToPickle.ClassToPickle, think of it like this:

When you pickled the class instance of ClassToPickle in the ClassToPickle.py script, the pickle module will pickle all the reference to the class which mean it will pickle the module name where the class was defined, and because you executed the script ClassToPickle.py this mean that the module will be set to __main__ which mean that pickle module will pickle __main__.ClassToPickle.

And when you tried to load the pickled instance it fail because it didn't find the instance's class which is __main__.ClassToPickle and not the one that you imported using from ClassToPickle import ClassToPickle because this latest is ClassToPickle.ClassToPickle.

A fix will be to create another script that will handle dumping instead of doing it in ClassToPickle.py e.g.

import pickle

from ClassToPickle import ClassToPickle

if __name__=="__main__":
    p = ClassToPickle(10)
    pickle.dump(p, open('10.pickle', 'w'))

namespace on python pickle

1 Answers