how to force ipython deep reload?

Question

I'm running a script in ipython via the run magic:

%run myscript.py

and this script imports various modules that I've written, and which I often change between runs. I want to automatically reload these modules, rather than having to restart ipython. There are lots of questions on stackoverflow and elsewhere that recommend

%load_ext autoreload
%autoreload 2

and maybe also

 %aimport <your module>

thrown in for good measure. But this simply doesn't work. Is a deep reload beyond what autoreload is capable of doing? Is there perhaps some other way of removing all loaded modules, or silently restart the python background process that ipython relies on?

EDIT: having played around with this a bit more, it seems that the autoreload failure is a little more subtle. It may be (I'm not 100% certain yet) that autoreload only fails when my module's __init__.py is doing from .<some file in the module> import *, rather than importing each member by name. I have also tried the %reset magic, but this only appears to empty the namespace, it doesn't clear the cached modules.

As an aside, Spyder is able to force reloading of modules, but I'm not sure how this is done (some kind of wrapper around ipython which restarts the process?)

funklute funklute · Accepted Answer · 2018-03-29T09:36:35

I looked up Spyder's solution, which basically uses del on sys.modules. Below I've slightly modified the code that I found in ~/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py, and I've put this in a module called reloader.py:

import sys


def _is_module_deletable(modname, modpath):
    if modname.startswith('_cython_inline'):
        # Don't return cached inline compiled .PYX files
        return False
    for path in [sys.prefix]:
        if modpath.startswith(path):
            return False
    else:
        return set(modname.split('.'))


def clear():
    """
    Del user modules to force Python to deeply reload them

    Do not del modules which are considered as system modules, i.e.
    modules installed in subdirectories of Python interpreter's binary
    Do not del C modules
    """
    log = []
    for modname, module in list(sys.modules.items()):
        modpath = getattr(module, '__file__', None)

        if modpath is None:
            # *module* is a C module that is statically linked into the
            # interpreter. There is no way to know its path, so we
            # choose to ignore it.
            continue

        if modname == 'reloader':
            # skip this module
            continue

        modules_to_delete = _is_module_deletable(modname, modpath)
        if modules_to_delete:
            log.append(modname)
            del sys.modules[modname]

    print("Reloaded modules:\n\n%s" % ", ".join(log))

now just do import reloader and then call reloader.clear()

how to force ipython deep reload?

1 Answers