CircleCI for simple Python project with tox: how to test multiple Python environments?

Question

I have a Python project for which I use tox to run the pytest-based tests. I am attempting to configure the project to build on CircleCI.

The tox.ini lists both Python 3.6 and 3.7 as environments:

envlist = py{36,37,},coverage

I can successfully run tox on a local machine within a conda virtual environment that uses Python version 3.7.

On CircleCI I am using a standard Python virtual environment since that is what is provided in the example ("getting started") configuration. The tox tests fail when tox attempts to create the Python 3.6 environment:

py36 create: /home/circleci/repo/.tox/py36
ERROR: InterpreterNotFound: python3.6

It appears that when you use this kind of virtual environment then tox can only find an interpreter of the same version, whereas if using a conda virtual environment it somehow knows how to cook up the environments as long as they're lower versions. At least for my case (Python 3.6 and 3.7 environments for tox running in a Python 3.7 conda environment), this works fine.

The CircleCI configuration file I'm currently using looks like this:

# Python CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-python/ for more details
#
version: 2
jobs:
  build:
    docker:
      # specify the version you desire here
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      - image: circleci/python:3.7

    working_directory: ~/repo

    steps:
      - checkout

      # Download and cache dependencies
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "requirements.txt" }}
            # fallback to using the latest cache if no exact match is found
            - v1-dependencies-

      - run:
          name: install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -e .
            pip install tox

      - save_cache:
          paths:
            - ./venv
          key: v1-dependencies-{{ checksum "requirements.txt" }}

      # run tests with tox
      - run:
          name: run tests
          command: |
            . venv/bin/activate
            tox

      - store_artifacts:
          path: test-reports
          destination: test-reports

What is the best practice for testing for multiple environments with tox on CircleCI? Should I move to using conda rather than venv within CircleCI, and if so how would I add this? Or is there a way to stay with venv, maybe by modifying its environment creation command?

Edit

I have now discovered that this is not specific to CircleCI, as I get a similar error when running this tox on Travis CI. Also, I have confirmed that this works as advertised using a Python 3.7 virtual environment created using venv on my local machine, both py36 and py37 environments run successfully.

James Adams James Adams · Accepted Answer · 2020-02-01T16:31:45

I have worked this out although not completely as it involves abandoning tox and instead using pytest for running tests for each Python version environment in separate workflow jobs. The CircleCI configuration (.circleci/config.yml) looks like this:

version: 2
workflows:
  version: 2
  test:
    jobs:
      - test-3.6
      - test-3.7
      - test-3.8
jobs:
  test-3.6: &test-template
    docker:
      - image: circleci/python:3.6
    working_directory: ~/repo
    steps:
      - checkout
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "requirements.txt" }}
          - v1-dependencies-
      - run:
          name: install dependencies
          command: |
            python3 -m venv venv
            . venv/bin/activate
            pip install -e .
            pip install coverage
            pip install pytest

      - save_cache:
          paths:
          - ./venv
          key: v1-dependencies-{{ checksum "requirements.txt" }}

      - run:
          name: run tests
          command: |
            . venv/bin/activate
            coverage run -m pytest tests

      # store artifacts (for example logs, binaries, etc)
      # to be available in the web app or through the API
      - store_artifacts:
          path: test-reports

  test-3.7:
    <<: *test-template
    docker:
      - image: circleci/python:3.7

  test-3.8:
    <<: *test-template
    docker:
      - image: circleci/python:3.8

CircleCI for simple Python project with tox: how to test multiple Python environments?

Edit

2 Answers