Coverage in tox for multiple python versions

Question

Here is a link to a project and output that you can use to reproduce the problem I describe below.

I'm using coverage with tox against multiple versions of python. My tox.ini file looks something like this:

[tox]
envlist =
    py27
    py34

[testenv]
deps =
    coverage

commands =
    coverage run --source=modules/ -m pytest
    coverage report -m

My problem is that coverage will run using only one version of python (in my case, py27), not both py27 and py34. This is a problem whenever I have code execution dependent on the python version, e.g.:

def add(a, b):
    import sys
    if sys.version.startswith('2.7'):
        print('2.7')
    if sys.version.startswith('3'):
        print('3')
    return a + b

Running coverage against the above code will incorrectly report that line 6 ("print('3')") is "Missing" for both py27 and py34. It should only be Missing for py34.

I know why this is happening: coverage is installed on my base OS (which uses python2.7). Thus, when tox is run, it notices that coverage is already installed and inherits coverage from the base OS rather than installing it in the virtualenv it creates.

This is fine and dandy for py27, but causes incorrect results in the coverage report for py34. I have a hacky, temporary work-around: I require a slightly earlier version of coverage (relative to the one installed on my base OS) so that tox will be forced to install a separate copy of coverage in the virtualenv. E.g.

[testenv]
deps =
    coverage==4.0.2
    pytest==2.9.0
    py==1.4.30

I don't like this workaround, but it's the best I've found for now. Any suggestions on a way to force tox to install the current version of coverage in its virtualenv's, even when I already have it installed on my base OS?

Answer 1

I came upon this problem today, but couldn't find an easy answer. So, for future reference, here is the solution that I came up with.

1. Create an envlist that contains each version of Python that will be tested and a custom env for cov.
2. For all versions of Python, set COVERAGE_FILE environment varible to store the .coverage file in {envdir}.
3. For the cov env I use two commands.
   1. coverage combine that combines the reports, and
   2. coverage html to generate the report and, if necessary, fail the test.
4. Create a .coveragerc file that contains a [paths] section to lists the source= locations.
   1. The first line is where the actual source code is found.
   2. The subsequent lines are the subpaths that will be eliminated by `coverage combine'.

tox.ini:

[tox]
envlist=py27,py36,py35,py34,py33,cov

[testenv]
deps=
    pytest
    pytest-cov
    pytest-xdist
setenv=
    py{27,36,35,34,33}: COVERAGE_FILE={envdir}/.coverage
commands=
    py{27,36,35,34,33}: python -m pytest --cov=my_project  --cov-report=term-missing --no-cov-on-fail
    cov: /usr/bin/env bash -c '{envpython} -m coverage combine {toxworkdir}/py*/.coverage'
    cov: coverage html --fail-under=85

.coveragerc:

[paths]
source=
    src/
    .tox/py*/lib/python*/site-packages/

The most peculiar part of the configuration is the invocation of coverage combine. Here's a breakdown of the command:

• tox does not handle Shell expansions {toxworkdir}/py*/.coverage, so we need to invoke a shell (bash -c) to get the necessary expansion.
  • If one were inclined, you could just type out all the paths individually and not jump through all of these hoops, but that would add maintenance and .coverage file dependency for each pyNN env.
• /usr/bin/env bash -c '...' to ensure we get the correct version of bash. Using the fullpath to env avoids the need for setting whitelist_externals.
• '{envpython} -m coverage ...' ensures that we invoke the correct python and coverage for the cov env.
• NOTE: The unfortunate problem of this solution is that the cov env is dependent on the invocation of py{27,36,35,34,33} which has some not so desirable side effects.
  • My suggestion would be to only invoke cov through tox.
  • Never invoke tox -ecov because, either
    • It will likely fail due to a missing .coverage file, or
    • It could give bizarre results (combining differing tests).
  • If you must invoke it as a subset (tox -epy27,py36,cov), then wipe out the .tox directory first (rm -rf .tox) to avoid the missing .coverage file problem.

Answer 2

I don't understand why tox wouldn't install coverage in each virtualenv properly. You should get two different coverage reports, one for py27 and one for py35. A nicer option might be to produce one combined report. Use coverage run -p to record separate data for each run, and then coverage combine to combine them before reporting.