What is the cleanest way to add a directory of third-party packages to the beginning of the Python path?

Question

My context is appengine_config.py, but this is really a general Python question.

Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:

dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):

sys.path.insert(1, dirpath)

However, that won't work if any of the packages in that directory are namespace packages.

To support namespace packages we can instead use:

site.addsitedir(dirpath)

But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.

The solution I have so far is this bit of code which I'd really like to simplify:

sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)

Is there a cleaner or more Pythonic way of accomplishing this?

My current version moves most of this code into an insertsitedir() function inside a vendor.py module, but otherwise hasn't really changed. Still looking for improvements or simplifications, if any are possible. — webmaven
After some more iterations, John Wayne Parrott has packaged up the resulting code: github.com/jonparrott/Darth-Vendor — webmaven

Remco Haszing Remco Haszing · Accepted Answer · 2014-09-07T14:08:54

For this answer I assume you know how to use setuptools and setup.py.

Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:

import os
import sys

if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
    # If we are unittesting, fake the non-existence of appengine_config.
    # The error message of the import error is handled by gae and must
    # exactly match the proper string.
    raise ImportError('No module named appengine_config')


# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
    for lib in os.listdir(lib_dir):
        if lib.endswith('.egg'):
            lib = os.path.join(lib_dir, lib)
            # Insert to override default libraries such as webob 1.1.1.
            sys.path.insert(0, lib)

And this piece of code in setup.cfg:

[develop]
install-dir = libs
always-copy = true

If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.

We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.

What is the cleanest way to add a directory of third-party packages to the beginning of the Python path?

2 Answers