python - Imports in __init__.py and `import as` statement

Question

I've run into a problem with having imports in __init__.py and using import as with absolute imports in modules of the package.

My project has a subpackage and in it's __init__.py I "lift" one of the classes from a module to the subpackage level with from import as statement. The module imports other modules from that subpackage with absolute imports. I get this error AttributeError: 'module' object has no attribute 'subpkg'.

Example

Structure:

pkg/
├── __init__.py
├── subpkg
│   ├── __init__.py
│   ├── one.py
│   └── two_longname.py
└── tst.py

pkg/__init__.py is empty.

pkg/subpkg/__init__.py:

from pkg.subpkg.one import One

pkg/subpkg/one.py:

import pkg.subpkg.two_longname as two

class One(two.Two):
    pass

pkg/subpkg/two_longname.py:

class Two:
    pass

pkg/tst.py:

from pkg.subpkg import One

print(One)

Output:

$ python3.4 -m pkg.tst
Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/and/dev/test/python/imptest2/pkg/tst.py", line 1, in <module>
    from pkg.subpkg import One
  File "/home/and/dev/test/python/imptest2/pkg/subpkg/__init__.py", line 1, in <module>
    from pkg.subpkg.one import One
  File "/home/and/dev/test/python/imptest2/pkg/subpkg/one.py", line 1, in <module>
    import pkg.subpkg.two_longname as two
AttributeError: 'module' object has no attribute 'subpkg'

Workarounds

There are changes that make it work:

1. Empty pkg/subpkg/__init__.py and importing directly from pkg.subpkg.one.

   I don't consider this as an option because AFAIK "lifting" things to the package level is ok. Here is quote from an article:

   One common thing to do in your __init__.py is to import selected Classes, functions, etc into the package level so they can be conveniently imported from the package.

2. Changing import as to from import in one.py:

   from pkg.subpkg import two_longname
   
   class One(two_longname.Two):
       pass
   

   The only con here is that I can't create a short alias for module. I got that idea from @begueradj's answer.

It is also possible to use a relative import in one.py to fix the problem. But I think it's just a variation of workaround #2.

Questions

1. Can someone explain what is actually going on here? Why a combination of imports in __init__.py and usage of import as leads to such problems?

2. Are there any better workarounds?

---

Original example

This is my original example. It's not very realistic but I'm not deleting it so @begueradj's answer still makes sense.

pkg/__init__.py is empty.

pkg/subpkg/__init__.py:

from pkg.subpkg.one import ONE

pkg/subpkg/one.py:

import pkg.subpkg.two
ONE = pkg.subpkg.two.TWO

pkg/subpkg/two.py:

TWO = 2

pkg/tst.py:

from pkg.subpkg import ONE

Output:

$ python3.4 -m pkg.tst
Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/and/dev/test/python/imptest/pkg/tst.py", line 1, in <module>
    from pkg.subpkg import ONE
  File "/home/and/dev/test/python/imptest/pkg/subpkg/__init__.py", line 2, in <module>
    from pkg.subpkg.one import ONE
  File "/home/and/dev/test/python/imptest/pkg/subpkg/one.py", line 6, in <module>
    ONE = pkg.subpkg.two.TWO
AttributeError: 'module' object has no attribute 'subpkg'

Initially I had this in one.py:

import pkg.subpkg.two as two
ONE = two.TWO

In that case I get error on import (just like in my original project which uses import as too).

Answer 1

You incorrectly assume that one cannot have an alias with from ... import, as from ... import ... as has been there since Python 2.0. The import ... as is the obscure syntax that not many know about, but which you use by accident in your code.

PEP 0221 claims that the following 2 are "effectively" the same:

1. import foo.bar.bazaar as baz
2. from foo.bar import bazaar as baz

The statement is not quite true in Python versions up to and including 3.6.x as evidenced by the corner case you met, namely if the required modules already exist in sys.modules but are yet uninitialized. The import ... as requires that the module foo.bar is injected in foo namespace as the attribute bar, in addition to being in sys.modules, whereas the from ... import ... as looks for foo.bar in sys.modules.

(Do note also that import foo.bar only ensures that the module foo.bar is in sys.modules and accessible as foo.bar, but might not be fully initialized yet.)

Changing the code as follows did the trick for me:

# import pkg.subpkg.two_longname as two
from pkg.subpkg import two_longname as two

And code runs perfectly on both Python 2 and Python 3.

Also, in one.py you cannot do from pkg import subpkg, for the same reason.

---

To demonstrate this bug further, fix your one.py as above, and add the following code in tst.py:

import pkg
import pkg.subpkg.two_longname as two

del pkg.subpkg

from pkg.subpkg import two_longname as two
import pkg.subpkg.two_longname as two

Only the last line crashes, because from ... import consults the sys.modules for pkg.subpkg and finds it there, whereas import ... as consults sys.modules for pkg and tries to find subpkg as an attribute in the pkg module. As we just had deleted that attribute, the last line fails with AttributeError: 'module' object has no attribute 'subpkg'.

---

As the import foo.bar as baz syntax is a bit obscure and adds more corner cases, and I have rarely if ever seen it being used, I would recommend avoiding it completely and favouring from .. import ... as.

Answer 2

Here is a theory on what's going on.

When you use the as reserved word, for instance:

import pkg.subpkg.two_longname as two

Python must to completely initialize and resolve all dependences that has to do with pkg.subpkg. But there is a problem, to completely load subpkg you need to completely load one.py as well right? wich at the same time imports two_longname.py using the as keyword ... Can you see the recursion here? That's why at the moment of doing:

import pkg.subpkg.two_longname as two

you get an error claiming subpkg does not exist.

To perform a test, go to one.py and change it to this:

#import pkg.subpkg.two_longname as two
from pkg.subpkg import two_longname

#class One(two.Two):
class One(two_longname.Two):
    pass

I suppose this is all about performance, Python loads a module partially whenever is possible. And the as keyword is one of the exceptions. I don't know if there are others, but it would be interesting know about them.

Answer 3

As the accepted answer states this is an issue with Python's behavior.

I've filed a bug: http://bugs.python.org/issue30024

The fix by Serhiy Storchaka was merged and expected in Python 3.7

Answer 4

Your project structure regarding the way you call modules, must be like this:

pkg/
├── __init__.py
├── subpkg
│   ├── __init__.py
│   ├── one.py
│   └── two.py
tst.py

Define your two.py like this:

class TWO:
    def functionTwo(self):
        print("2")

Define your one.py like this :

from pkg.subpkg import two
class ONE:
    def functionOne(self):
        print("1")

        self.T=two.TWO()
        print("Calling TWO from ONE: ")
        self.T.functionTwo()

Define your test.py like this

from pkg.subpkg import one
class TEST:
    def functionTest(self):
        O=one.ONE()
        O.functionOne()
if __name__=='__main__':
    T=TEST()
    T.functionTest()

When you execute, you will get this:

1
Calling  TWO from  ONE:
2

Hope this helps.