Reloading packages (and their submodules) recursively in Python

13
votes

In Python you can reload a module as follows...

import foobar

import importlib
importlib.reload(foobar)

This works for .py files, but for Python packages it will only reload the package and not any of the nested sub-modules.

With a package:

  • foobar/__init__.py
  • foobar/spam.py
  • foobar/eggs.py

Python Script:

import foobar

# assume `spam/__init__.py` is importing `.spam`
# so we dont need an explicit import.
print(foobar.spam)  # ok

import importlib
importlib.reload(foobar)
# foobar.spam WONT be reloaded.

Not to suggest this is a bug, but there are times its useful to reload a package and all its submodules. (If you want to edit a module while a script runs for example).

What are some good ways to recursively reload a package in Python?

Notes:

  • For the purpose of this question assume the latest Python3.x

    (currently using importlib)

  • Allowing that this may requre some edits to the modules themselves.
  • Assume that wildcard imports aren't used (from foobar import *), since they may complicate reload logic.
2
pythonpython-3.x
IPython provides the IPython.lib.deepreload module for recursive reloading. The code can be found here. Interestingly, the module is 285 sloc. – jme
Great hint, but replacing import hooks is a kind of heavy weight solution (which makes sense for IPython) but not some snippet I would want in my project just to get reloading working a little more usefully. – ideasman42

2 Answers

6
votes

Heres a function that recursively loads a package. Double checked that the reloaded modules are updated in the modules where they are used, and that issues with infinite recursion are checked for.

One restruction is it needs to run on a package (which only makes sense for packages anyway)

import os
import types
import importlib


def reload_package(package):
    assert(hasattr(package, "__package__"))
    fn = package.__file__
    fn_dir = os.path.dirname(fn) + os.sep
    module_visit = {fn}
    del fn

    def reload_recursive_ex(module):
        importlib.reload(module)

        for module_child in vars(module).values():
            if isinstance(module_child, types.ModuleType):
                fn_child = getattr(module_child, "__file__", None)
                if (fn_child is not None) and fn_child.startswith(fn_dir):
                    if fn_child not in module_visit:
                        # print("reloading:", fn_child, "from", module)
                        module_visit.add(fn_child)
                        reload_recursive_ex(module_child)

    return reload_recursive_ex(package)

# example use
import os
reload_package(os)
0
votes

I'll offer another answer for the case in which you want to reload only a specific nested module. I found this to be useful for situations where I found myself editing a single subnested module, and reloading all sub-nested modules via a solution like ideasman42's approach or deepreload would produce undesired behavior.

assuming you want to reload a module into the workspace below

my_workspace.ipynb

import importlib
import my_module
import my_other_module_that_I_dont_want_to_reload

print(my_module.test()) #old result
importlib.reload(my_module)
print(my_module.test()) #new result

but my_module.py looks like this:

import my_nested_submodule

def test():
   my_nested_submodule.do_something()

and you just made an edit in my_nested_submodule.py:

def do_something():
   print('look at this cool new functionality!')

You can manually force my_nested_submodule, and only my_nested_submodule to be reloaded by adjusting my_module.py so it looks like the following:

import my_nested_submodule
import importlib
importlib.reload(my_nested_submodule)

def test():
   my_nested_submodule.do_something()