How to use a local repository with pip install for a package that uses setup_requires within dh_virtualenv

Question

(Sorry this one is a little bit long, but I want to describe my problem and what I found out so far)

I am dependent on some python packages (jira, PyNaCl). These use setup_requires which I seem not to be able to control with dh_virtualenv. My problem is that I have to build the whole thing from source using a local mini-buildd. While building I have no connection to PyPi.python.org and I don't have a local proxy available. If I could control easy_install I could try to point it to my local repository, which I include into my debian source package.

But here lies my problem: setup_requires (distutils) ignores the parameter I pass to pip for the local repository. (See pip.pypa.io for that. Mini-buildd builds a completely new chroot for each build so I don't know how to add extra lines to any of the Distutils Configuration Files. Without this I end up at an error message like the following:

Running setup.py (path:/tmp/pip-build-vSvelX/PyNaCl/setup.py) egg_info for package PyNaCl
Running command python setup.py egg_info
Download error on https://pypi.python.org/simple/cffi/: [Errno 110] Connection timed out -- Some packages may not be found!
Couldn't find index page for 'cffi' (maybe misspelled?)

So what have I managed to build up so far?

There is a local repository (./requirements) with all the tar.gz files that my package relies on (pip freeze, pip download --no-binary :all:) included in the debian source package.

pip install -vvv --isolated--ignore-installed --no-cache-dir --no-deps --no-index --find-links=./requirements -r requirements.txt

in a new virtualenv builds just fine as long as I have either:

• A connection to pypi.
• A working config file for Distutils

Removing the config file and the the internet connection e.g. via

export HTTPS_PROXY=invalid; export http_proxy=invalid

the above pip command will abort with a message like:

Running setup.py (path:/tmp/pip-build-88bVhg/PyNaCl/setup.py) egg_info for package PyNaCl
Running command python setup.py egg_info
Download error on https://pypi.python.org/simple/pycparser/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
Couldn't find index page for 'pycparser' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno -3] Temporary failure in name resolution -- Some packages may not be found!
No local packages or working download links found for pycparser

To fix it I can either:

1. enter into the breaking tar-balls, execute python setup.py egg_info, build a new tar ball and upload the new tarball
2. add $HOME/.pydistutils.cfg containing the an easy_install configuration.

Why is this not satisfactory for me?

1. building egg.info in the respective tarballs would mean that I have to touch packages I do not maintain and updating could be "interesting"
2. using configuration files don't seem to work for me because:
   • The python directory is set up fresh during the build-process
   • I don't have a Home-directory on the mini-buildd
   • To modify setup.cfg I would have to have advanced knowledge where my main package will end up. And I would have to manipulate foreign packages.
3. Getting a network connection to a PyPi cache or mirror at build time is not an option either

So my question is if there is another way to get the configuration options to easy_install or to make pip install ignore the egg_info part all together while still installing the direct requirements into the virtualenv for dh_virtualenv?

Answer 1

I found a way around this. Does not do what I requested in the question but solves the underlying problem.

dh_virtualenv provides the option "--preinstall". Using this I can build without any network connection.

So I added the following to debian/rules:

override_dh_virtualenv --preinstall '--requirement=./requirements-preinstall.txt'

/requirements-preinstall.txt contains the packages that distutils tried to download and failed. Somehow it seems to work out if there are multiple install rounds.