poetry dependencies not available when running script with tox

3
votes

I have a python project that uses poetry and tox. It has source code, tests and scripts (juptext notebooks). I can't import the dev dependencies in the scripts, but I can in the tests.

When I came across this problem, I created the following minimal example. At first, it didn't work, then I fiddled around with it, and now it's working. So I stripped the project that has the actual problem down so it's indistinguishable other than the project name, location, virtual env, and .git directory, but that's still not working.

UPDATE deleting all build artifacts and the virtualenv for the minimal example makes it stop working again

UPDATE adding the line scripts: poetry install to the tox commands fixed only the minimal example

The source code, tests and scripts are in the following layout

foo
  +--foo
  |  +--__init__.py
  |
  +--tests
  |  +--__init__.py
  |  +--test_foo.py
  |
  +--scripts
  |  +--foo_script.py
  |
  +--pyproject.toml
  +--tox.ini

The files are either empty or as follows:

foo_script.py

import requests

test_foo.py

import requests
import pytest

def test():
    assert True

pyproject.toml

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo maker"]

[tool.poetry.dependencies]
python = "^3.7"
requests = "*"

[tool.poetry.dev-dependencies]
pytest = "^4.6"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

tox.ini

[tox]
envlist = test, scripts
isolated_build = true
skipsdist = true

[testenv]
basepython = python3.7
whitelist_externals =
    pytest
    bash
commands =
    test: pytest
    scripts: bash -c 'python3 scripts/*.py'

When I run tox, I get

test run-test-pre: PYTHONHASHSEED='4126239415'
test run-test: commands[0] | pytest
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
cachedir: .tox/test/.pytest_cache
rootdir: /home/#######/foo
collected 1 item                                                               

tests/test_foo.py .                                                      [100%]

============================== 1 passed in 0.09s ===============================
scripts run-test-pre: PYTHONHASHSEED='4126239415'
scripts run-test: commands[0] | bash -c 'python3 scripts/*.py'
Traceback (most recent call last):
  File "scripts/foo_script.py", line 1, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
ERROR: InvocationError for command /bin/bash -c 'python3 scripts/*.py' (exited with code 1)
___________________________________ summary ____________________________________
  test: commands succeeded
ERROR:   scripts: commands failed
3
pythontoxpython-poetry
Do you have pytest installed globally? The tox testenv named test is supposed to call the command pytest, but it is not declared anywhere as a dependency as far as I can tell. So this testenv should fail. – sinoroc
@sinoroc apparently so. wasn't aware of that – joel
As far as I can tell, tox can't recognize poetry's dev-dependencies (I am not sure why poetry invented its own thing here). A more common pattern that you might want to follow is to use a test extra instead to make sure that test dependencies such as pytest are installed in the _tox environments, like explained in this answer: stackoverflow.com/a/59522588/11138259 – sinoroc

3 Answers

0
votes

I believe something like the following should work:

pyproject.toml

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo maker"]

[tool.poetry.dependencies]
python = "^3.7"
requests = "*"
#
pytest = { version = "^4.6", optional = true }

[tool.poetry.extras]
test = ["pytest"]

# [tool.poetry.dev-dependencies]
# use 'test' extra instead

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

tox.ini

[tox]
envlist = test, scripts
isolated_build = true

[testenv]
basepython = python3.7
whitelist_externals =
    pytest
    bash
extras =
    test
commands =
    test: pytest
    scripts: bash -c 'for f in scripts/*.py; do python "$f"; done'
0
votes

Assuming you have installed poetry and tox and pytest are dependencies in your pyproject.yml (notice poetry run, see https://python-poetry.org/docs/cli/#run):

[tox]
envlist = py37
isolated_build = True
skipsdist = True

[testenv]
whitelist_externals = poetry
commands=
    poetry run pytest

optionally you can make the install happen on running the tests by changing the last bit to (but then you will need to have tox installed outside of poetry, which could cause you issues down the line)

commands=
    poetry install
    poetry run pytest

Also depending on your root folder and where the tests are you can configure the path for tox to change directory to by adding

changedir = tests

In which case the whole file would look like this if you are in directory foo executing tox:

[tox]
envlist = py37
isolated_build = True
skipsdist = True

[testenv]
whitelist_externals = poetry
commands=
    poetry run pytest
changedir = tests
-1
votes

First I need to install the dependencies with poetry install. Then append poetry run to the beginning of the command to enable the dependencies. Also running python scripts like that will just run the first, passing the name of the others as args to the first program. Instead use for f in scripts/*.py; do python "$f"; done (see here)

All together

poetry install
poetry run bash -c 'for f in scripts/*.py; do python "$f"; done'