Relative imports in Python 3

Question

I want to import a function from another file in the same directory.

Sometimes it works for me with from .mymodule import myfunction but sometimes I get a:

SystemError: Parent module '' not loaded, cannot perform relative import

Sometimes it works with from mymodule import myfunction, but sometimes I also get a:

SystemError: Parent module '' not loaded, cannot perform relative import

I don't understand the logic here, and I couldn't find any explanation. This looks completely random.

Could someone explain to me what's the logic behind all this?

Answer 1

unfortunately, this module needs to be inside the package, and it also needs to be runnable as a script, sometimes. Any idea how I could achieve that?

It's quite common to have a layout like this...

main.py
mypackage/
    __init__.py
    mymodule.py
    myothermodule.py

...with a mymodule.py like this...

#!/usr/bin/env python3

# Exported function
def as_int(a):
    return int(a)

# Test function for module  
def _test():
    assert as_int('1') == 1

if __name__ == '__main__':
    _test()

...a myothermodule.py like this...

#!/usr/bin/env python3

from .mymodule import as_int

# Exported function
def add(a, b):
    return as_int(a) + as_int(b)

# Test function for module  
def _test():
    assert add('1', '1') == 2

if __name__ == '__main__':
    _test()

...and a main.py like this...

#!/usr/bin/env python3

from mypackage.myothermodule import add

def main():
    print(add('1', '1'))

if __name__ == '__main__':
    main()

...which works fine when you run main.py or mypackage/mymodule.py, but fails with mypackage/myothermodule.py, due to the relative import...

from .mymodule import as_int

The way you're supposed to run it is...

python3 -m mypackage.myothermodule

...but it's somewhat verbose, and doesn't mix well with a shebang line like #!/usr/bin/env python3.

The simplest fix for this case, assuming the name mymodule is globally unique, would be to avoid using relative imports, and just use...

from mymodule import as_int

...although, if it's not unique, or your package structure is more complex, you'll need to include the directory containing your package directory in PYTHONPATH, and do it like this...

from mypackage.mymodule import as_int

...or if you want it to work "out of the box", you can frob the PYTHONPATH in code first with this...

import sys
import os

PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))

from mypackage.mymodule import as_int

It's kind of a pain, but there's a clue as to why in an email written by a certain Guido van Rossum...

I'm -1 on this and on any other proposed twiddlings of the __main__ machinery. The only use case seems to be running scripts that happen to be living inside a module's directory, which I've always seen as an antipattern. To make me change my mind you'd have to convince me that it isn't.

Whether running scripts inside a package is an antipattern or not is subjective, but personally I find it really useful in a package I have which contains some custom wxPython widgets, so I can run the script for any of the source files to display a wx.Frame containing only that widget for testing purposes.

Answer 2

Explanation

From PEP 328

Relative imports use a module's __name__ attribute to determine that module's position in the package hierarchy. If the module's name does not contain any package information (e.g. it is set to '__main__') then relative imports are resolved as if the module were a top level module, regardless of where the module is actually located on the file system.

At some point PEP 338 conflicted with PEP 328:

... relative imports rely on __name__ to determine the current module's position in the package hierarchy. In a main module, the value of __name__ is always '__main__', so explicit relative imports will always fail (as they only work for a module inside a package)

and to address the issue, PEP 366 introduced the top level variable __package__:

By adding a new module level attribute, this PEP allows relative imports to work automatically if the module is executed using the -m switch. A small amount of boilerplate in the module itself will allow the relative imports to work when the file is executed by name. [...] When it [the attribute] is present, relative imports will be based on this attribute rather than the module __name__ attribute. [...] When the main module is specified by its filename, then the __package__ attribute will be set to None. [...] When the import system encounters an explicit relative import in a module without __package__ set (or with it set to None), it will calculate and store the correct value (__name__.rpartition('.')[0] for normal modules and __name__ for package initialisation modules)

(emphasis mine)

If the __name__ is '__main__', __name__.rpartition('.')[0] returns empty string. This is why there's empty string literal in the error description:

SystemError: Parent module '' not loaded, cannot perform relative import

The relevant part of the CPython's PyImport_ImportModuleLevelObject function:

if (PyDict_GetItem(interp->modules, package) == NULL) {
    PyErr_Format(PyExc_SystemError,
            "Parent module %R not loaded, cannot perform relative "
            "import", package);
    goto error;
}

CPython raises this exception if it was unable to find package (the name of the package) in interp->modules (accessible as sys.modules). Since sys.modules is "a dictionary that maps module names to modules which have already been loaded", it's now clear that the parent module must be explicitly absolute-imported before performing relative import.

Note: The patch from the issue 18018 has added another if block, which will be executed before the code above:

if (PyUnicode_CompareWithASCIIString(package, "") == 0) {
    PyErr_SetString(PyExc_ImportError,
            "attempted relative import with no known parent package");
    goto error;
} /* else if (PyDict_GetItem(interp->modules, package) == NULL) {
    ...
*/

If package (same as above) is empty string, the error message will be

ImportError: attempted relative import with no known parent package

However, you will only see this in Python 3.6 or newer.

Solution #1: Run your script using -m

Consider a directory (which is a Python package):

.
├── package
│   ├── __init__.py
│   ├── module.py
│   └── standalone.py

All of the files in package begin with the same 2 lines of code:

from pathlib import Path
print('Running' if __name__ == '__main__' else 'Importing', Path(__file__).resolve())

I'm including these two lines only to make the order of operations obvious. We can ignore them completely, since they don't affect the execution.

__init__.py and module.py contain only those two lines (i.e., they are effectively empty).

standalone.py additionally attempts to import module.py via relative import:

from . import module  # explicit relative import

We're well aware that /path/to/python/interpreter package/standalone.py will fail. However, we can run the module with the -m command line option that will "search sys.path for the named module and execute its contents as the __main__ module":

vaultah@base:~$ python3 -i -m package.standalone
Importing /home/vaultah/package/__init__.py
Running /home/vaultah/package/standalone.py
Importing /home/vaultah/package/module.py
>>> __file__
'/home/vaultah/package/standalone.py'
>>> __package__
'package'
>>> # The __package__ has been correctly set and module.py has been imported.
... # What's inside sys.modules?
... import sys
>>> sys.modules['__main__']
<module 'package.standalone' from '/home/vaultah/package/standalone.py'>
>>> sys.modules['package.module']
<module 'package.module' from '/home/vaultah/package/module.py'>
>>> sys.modules['package']
<module 'package' from '/home/vaultah/package/__init__.py'>

-m does all the importing stuff for you and automatically sets __package__, but you can do that yourself in the

Solution #2: Set __package__ manually

Please treat it as a proof of concept rather than an actual solution. It isn't well-suited for use in real-world code.

PEP 366 has a workaround to this problem, however, it's incomplete, because setting __package__ alone is not enough. You're going to need to import at least N preceding packages in the module hierarchy, where N is the number of parent directories (relative to the directory of the script) that will be searched for the module being imported.

Thus,

1. Add the parent directory of the Nth predecessor of the current module to sys.path

2. Remove the current file's directory from sys.path

3. Import the parent module of the current module using its fully-qualified name

4. Set __package__ to the fully-qualified name from 2

5. Perform the relative import

I'll borrow files from the Solution #1 and add some more subpackages:

package
├── __init__.py
├── module.py
└── subpackage
    ├── __init__.py
    └── subsubpackage
        ├── __init__.py
        └── standalone.py

This time standalone.py will import module.py from the package package using the following relative import

from ... import module  # N = 3

We'll need to precede that line with the boilerplate code, to make it work.

import sys
from pathlib import Path

if __name__ == '__main__' and __package__ is None:
    file = Path(__file__).resolve()
    parent, top = file.parent, file.parents[3]

    sys.path.append(str(top))
    try:
        sys.path.remove(str(parent))
    except ValueError: # Already removed
        pass

    import package.subpackage.subsubpackage
    __package__ = 'package.subpackage.subsubpackage'

from ... import module # N = 3

It allows us to execute standalone.py by filename:

vaultah@base:~$ python3 package/subpackage/subsubpackage/standalone.py
Running /home/vaultah/package/subpackage/subsubpackage/standalone.py
Importing /home/vaultah/package/__init__.py
Importing /home/vaultah/package/subpackage/__init__.py
Importing /home/vaultah/package/subpackage/subsubpackage/__init__.py
Importing /home/vaultah/package/module.py

A more general solution wrapped in a function can be found here. Example usage:

if __name__ == '__main__' and __package__ is None:
    import_parents(level=3) # N = 3

from ... import module
from ...module.submodule import thing

Solution #3: Use absolute imports and setuptools

The steps are -

1. Replace explicit relative imports with equivalent absolute imports

2. Install package to make it importable

For instance, the directory structure may be as follows

.
├── project
│   ├── package
│   │   ├── __init__.py
│   │   ├── module.py
│   │   └── standalone.py
│   └── setup.py

where setup.py is

from setuptools import setup, find_packages
setup(
    name = 'your_package_name',
    packages = find_packages(),
)

The rest of the files were borrowed from the Solution #1.

Installation will allow you to import the package regardless of your working directory (assuming there'll be no naming issues).

We can modify standalone.py to use this advantage (step 1):

from package import module  # absolute import

Change your working directory to project and run /path/to/python/interpreter setup.py install --user (--user installs the package in your site-packages directory) (step 2):

vaultah@base:~$ cd project
vaultah@base:~/project$ python3 setup.py install --user

Let's verify that it's now possible to run standalone.py as a script:

vaultah@base:~/project$ python3 -i package/standalone.py
Running /home/vaultah/project/package/standalone.py
Importing /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/__init__.py
Importing /home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py
>>> module
<module 'package.module' from '/home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py'>
>>> import sys
>>> sys.modules['package']
<module 'package' from '/home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/__init__.py'>
>>> sys.modules['package.module']
<module 'package.module' from '/home/vaultah/.local/lib/python3.6/site-packages/your_package_name-0.0.0-py3.6.egg/package/module.py'>

Note: If you decide to go down this route, you'd be better off using virtual environments to install packages in isolation.

Solution #4: Use absolute imports and some boilerplate code

Frankly, the installation is not necessary - you could add some boilerplate code to your script to make absolute imports work.

I'm going to borrow files from Solution #1 and change standalone.py:

1. Add the parent directory of package to sys.path before attempting to import anything from package using absolute imports:

   import sys
   from pathlib import Path # if you haven't already done so
   file = Path(__file__).resolve()
   parent, root = file.parent, file.parents[1]
   sys.path.append(str(root))
   
   # Additionally remove the current file's directory from sys.path
   try:
       sys.path.remove(str(parent))
   except ValueError: # Already removed
       pass
   

2. Replace the relative import by the absolute import:

   from package import module  # absolute import
   

standalone.py runs without problems:

vaultah@base:~$ python3 -i package/standalone.py
Running /home/vaultah/package/standalone.py
Importing /home/vaultah/package/__init__.py
Importing /home/vaultah/package/module.py
>>> module
<module 'package.module' from '/home/vaultah/package/module.py'>
>>> import sys
>>> sys.modules['package']
<module 'package' from '/home/vaultah/package/__init__.py'>
>>> sys.modules['package.module']
<module 'package.module' from '/home/vaultah/package/module.py'>

I feel that I should warn you: try not to do this, especially if your project has a complex structure.

---

As a side note, PEP 8 recommends the use of absolute imports, but states that in some scenarios explicit relative imports are acceptable:

Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages). [...] However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose.

Answer 3

Put this inside your package's __init__.py file:

# For relative imports to work in Python 3.6
import os, sys; sys.path.append(os.path.dirname(os.path.realpath(__file__)))

Assuming your package is like this:

├── project
│   ├── package
│   │   ├── __init__.py
│   │   ├── module1.py
│   │   └── module2.py
│   └── setup.py

Now use regular imports in you package, like:

# in module2.py
from module1 import class1

This works in both python 2 and 3.

Answer 4

I ran into this issue. A hack workaround is importing via an if/else block like follows:

#!/usr/bin/env python3
#myothermodule

if __name__ == '__main__':
    from mymodule import as_int
else:
    from .mymodule import as_int


# Exported function
def add(a, b):
    return as_int(a) + as_int(b)

# Test function for module  
def _test():
    assert add('1', '1') == 2

if __name__ == '__main__':
    _test()

Answer 5

For PyCharm users:

I also was getting ImportError: attempted relative import with no known parent package because I was adding the . notation to silence a PyCharm parsing error. PyCharm innaccurately reports not being able to find:

lib.thing import function

If you change it to:

.lib.thing import function

it silences the error but then you get the aforementioned ImportError: attempted relative import with no known parent package. Just ignore PyCharm's parser. It's wrong and the code runs fine despite what it says.

Answer 6

To obviate this problem, I devised a solution with the repackage package, which has worked for me for some time. It adds the upper directory to the lib path:

import repackage
repackage.up()
from mypackage.mymodule import myfunction

Repackage can make relative imports that work in a wide range of cases, using an intelligent strategy (inspecting the call stack).

Answer 7

Hopefully, this will be of value to someone out there - I went through half a dozen stackoverflow posts trying to figure out relative imports similar to whats posted above here. I set up everything as suggested but I was still hitting ModuleNotFoundError: No module named 'my_module_name'

Since I was just developing locally and playing around, I hadn't created/run a setup.py file. I also hadn't apparently set my PYTHONPATH.

I realized that when I ran my code as I had been when the tests were in the same directory as the module, I couldn't find my module:

$ python3 test/my_module/module_test.py                                                                                                               2.4.0
Traceback (most recent call last):
  File "test/my_module/module_test.py", line 6, in <module>
    from my_module.module import *
ModuleNotFoundError: No module named 'my_module'

However, when I explicitly specified the path things started to work:

$ PYTHONPATH=. python3 test/my_module/module_test.py                                                                                                  2.4.0
...........
----------------------------------------------------------------------
Ran 11 tests in 0.001s

OK

So, in the event that anyone has tried a few suggestions, believes their code is structured correctly and still finds themselves in a similar situation as myself try either of the following if you don't export the current directory to your PYTHONPATH:

1. Run your code and explicitly include the path like so: $ PYTHONPATH=. python3 test/my_module/module_test.py
2. To avoid calling PYTHONPATH=., create a setup.py file with contents like the following and run python setup.py development to add packages to the path:

# setup.py
from setuptools import setup, find_packages

setup(
    name='sample',
    packages=find_packages()
)

Answer 8

I needed to run python3 from the main project directory to make it work.

For example, if the project has the following structure:

project_demo/
├── main.py
├── some_package/
│   ├── __init__.py
│   └── project_configs.py
└── test/
    └── test_project_configs.py

Solution

I would run python3 inside folder project_demo/ and then perform a

from some_package import project_configs

Answer 9

I was getting this ImportError: attempted relative import with no known parent package

In my program I was using the file from current path for importing its function.

from .filename import function

Then I modified the current path (Dot) with package name. Which resolved my issue.

from package_name.filename import function

I hope the above answer helps you.

Answer 10

I tried all of the above to no avail, only to realize I mistakenly had a - in my package name.

In short, don't have - in the directory where __init__.py is. I've never felt elated after finding out such inanity.

Answer 11

My boilerplate to make a module with relative imports in a package runnable standalone.

package/module.py

## Standalone boilerplate before relative imports
if __package__ is None:                  
    DIR = Path(__file__).resolve().parent
    sys.path.insert(0, str(DIR.parent))
    __package__ = DIR.name

from . import variable_in__init__py
from . import other_module_in_package
...

Now you can use your module in any fashion:

1. Run module as usual: python -m package.module
2. Use it as a module: python -c 'from package import module'
3. Run it standalone: python package/module.py
4. or with shebang (#!/bin/env python) just: package/module.py

NB! Using sys.path.append instead of sys.path.insert will give you a hard to trace error if your module has the same name as your package. E.g. my_script/my_script.py

Of course if you have relative imports from higher levels in your package hierarchy, than this is not enough, but for most cases, it's just okay.

Answer 12

if both packages are in your import path (sys.path), and the module/class you want is in example/example.py, then to access the class without relative import try:

from example.example import fkt

Answer 13

If none of the above worked for you, you can specify the module explicitly.

Directory:

├── Project
│     ├── Dir
│     │    ├── __init__.py
│     │    ├── module.py
│     │    └── standalone.py

Solution:

#in standalone.py
from Project.Dir.module import ...

module - the module to be imported

Answer 14

I think the best solution is to create a package for your module: Here is more info on how to do it.

Once you have a package you don't need to worry about relative import, you can just do absolute imports.

Answer 15

I had a similar problem: I needed a Linux service and cgi plugin which use common constants to cooperate. The 'natural' way to do this is to place them in the init.py of the package, but I cannot start the cgi plugin with the -m parameter.

My final solution was similar to Solution #2 above:

import sys
import pathlib as p
import importlib

pp = p.Path(sys.argv[0])
pack = pp.resolve().parent

pkg = importlib.import_module('__init__', package=str(pack))

The disadvantage is that you must prefix the constants (or common functions) with pkg:

print(pkg.Glob)

Answer 16

Moving the file from which you are importing to an outside directory helps.
This is extra useful when your main file makes any other files in its own directory.
Ex:
Before:
Project
|---dir1
|-------main.py
|-------module1.py
After:
Project
|---module1.py
|---dir1
|-------main.py

Answer 17

TLDR; Append Script path to the System Path by adding following in the entry point of your python script.

import os.path
import sys
PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))

Thats it now you can run your project in PyCharma as well as from Terminal!!

Answer 18

I encounter this a lot when I am working with Django, since a lot of functionality is performed from the manage.py script but I also want to have some of my modules runnable directly as scripts as well (ideally you would make them manage.py directives but we're not there yet).

This is a mock up of what such a project might look like;

├── dj_app
│   ├── models.py
│   ├── ops
│   │   ├── bar.py
│   │   └── foo.py
│   ├── script.py
│   ├── tests.py
│   ├── utils.py
│   └── views.py
└── manage.py

The important parts here being manage.py, dj_app/script.py, and dj_app/tests.py. We also have submodules dj_app/ops/bar.py and dj_app/ops/foo.py which contain more items we want to use throughout the project.

The source of the issue commonly comes from wanting your dj_app/script.py script methods to have test cases in dj_app/tests.py which get invoked when you run manage.py test.

This is how I set up the project and its imports;

# dj_app/ops/foo.py
# Foo operation methods and classes
foo_val = "foo123"

.

# dj_app/ops/bar.py
# Bar operations methods and classes
bar_val = "bar123"

.

# dj_app/script.py
# script to run app methods from CLI

# if run directly from command line
if __name__ == '__main__':
    from ops.bar import bar_val
    from ops.foo import foo_val

# otherwise
else:
    from .ops.bar import bar_val
    from .ops.foo import foo_val

def script_method1():
    print("this is script_method1")
    print("bar_val: {}".format(bar_val))
    print("foo_val: {}".format(foo_val))


if __name__ == '__main__':
    print("running from the script")
    script_method1()

.

# dj_app/tests.py
# test cases for the app
# do not run this directly from CLI or the imports will break
from .script import script_method1
from .ops.bar import bar_val
from .ops.foo import foo_val 

def main():
    print("Running the test case")
    print("testing script method")
    script_method1()

if __name__ == '__main__':
    print("running tests from command line")
    main()

.

# manage.py
# just run the test cases for this example
import dj_app.tests
dj_app.tests.main()

.

Running the test cases from manage.py;

$ python3 manage.py
Running the test case
testing script method
this is script_method1
bar_val: bar123
foo_val: foo123

Running the script on its own;

$ python3 dj_app/script.py
running from the script
this is script_method1
bar_val: bar123
foo_val: foo123

Note that you get an error if you try to run the test.py directly however, so don't do that;

$ python3 dj_app/tests.py
Traceback (most recent call last):
  File "dj_app/tests.py", line 5, in <module>
    from .script import script_method1
ModuleNotFoundError: No module named '__main__.script'; '__main__' is not a package

If I run into more complicated situations for imports, I usually end up implementing something like this to hack through it;

import os
import sys
THIS_DIR = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, THIS_DIR)
from script import script_method1
sys.path.pop(0)

Answer 19

This my project structure

├── folder
|   | 
│   ├── moduleA.py
|   |   |
|   |   └--function1()
|   |       └~~ uses function2()
|   | 
│   └── moduleB.py
|       | 
|       └--function2()
|   
└── main.py
     └~~ uses function1()

Here my moduleA imports moduleB and main imports moduleA

I added the snippet below in moduleA to import moduleB

try:
    from .moduleB import function2 
except:
    from moduleB import function2 

Now I can execute both main.py as well as moduleA.py individually

Is this a solution ?

Answer 20

I had a similar problem and solved it by creating a symbolic link to the package in the working directory:

ln -s ../../../my_package my_package

and then import it as usual:

import my_package

I know this is more like a "Linux" solution rather than a "Python" solution. but it's a valid approach nonetheless.