python - Test case execution order in pytest

Question

I am using pytest. I have two files in a directory. In one of the files there is a long running test case that generates some output. In the other file there is a test case that reads that output. How can I ensure the proper execution order of the two test cases? Is there any alternative other than puting the test cases in the same file in the proper order?

Answer 1

In general you can configure the behavior of basically any part of pytest using its well-specified hooks.

In your case, you want the "pytest_collection_modifyitems" hook, which lets you re-order collected tests in place.

That said, it does seem like ordering your tests should be easier -- this is Python after all! So I wrote a plugin for ordering tests: "pytest-ordering". Check out the docs or install it from pypi. Right now I recommend using @pytest.mark.first and @pytest.mark.second, or one of the @pytest.mark.order# markers, but I have some ideas about more useful APIs. Suggestions welcome :)

Edit: pytest-ordering seems abandoned at the moment, you can also check out pytest-order (a fork of the original project by the author).

Answer 2

Maybe you can consider using dependency pytest plugin, where you can set the test dependencies easily.

Be careful - the comments suggest this does not work for everyone.

@pytest.mark.dependency()
def test_long():
    pass

@pytest.mark.dependency(depends=['test_long'])
def test_short():
    pass

This way test_short will only execute if test_long is success and force the execution sequence as well.

Answer 3

There's also a plugin pytest-ordering that seems to meet your requirements.

Answer 4

main.py:

import functools
import pytest
from demo import test_foo,test_hi

def check_depends(depends):
    try:
        for dep in depends:
            dep()
    except Exception as e:
        return dep
    else:
        return True

def pytest_depend(depends):
    def pytest_depend_decorator(func):
        stat = check_depends(depends)
        if stat is True:
            return func
        else:
            return pytest.mark.skip(True, reason="%s[skip] --> %s[Failed]" % (func.__name__, stat.__name__))(func)
    return pytest_depend_decorator


@pytest_depend([test_foo,test_hi])
def test_bar():
    pass

@pytest_depend([test_foo,test_hi])
def test_bar2():
    pass

demo.py:

def test_hi():
    pass
def test_foo():
    assert False

---

platform linux -- Python 3.5.2, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /usr/bin/python3

pytest -vrsx ./plugin.py

Answer 5

It's important to keep in mind, while trying to fix pytest ordering "issue", that running tests in the same order as they are specified seems to be the default of pytest.

It turns out that my tests were out of that order because of one of these packages - pytest-dependency, pytest-depends, pytest-order. Once I uninstalled them all with pip uninstall package_name, the problem was gone. Looks like they have side effects

Answer 6

Try this:

@pytest.fixture(xxx)
def test_A():
    pass
    yield
    pass

@pytest.mark.usefixtures('test_A')
def test_B():
    pass

Answer 7

Make use of the '--randomly-dont-reorganize' option or '-p no:randomly' available in pytest-randomly plugin, this will just run your test in the same order as you mentioned in your module.

Module:

import pytest

def test_three():
    assert True

def test_four():
    assert True

def test_two():
    assert True

def test_one():
    assert True

Execution:

(tmp.w95BqE188N) rkalaiselvan@dev-rkalaiselvan:~/$ py.test --randomly-dont-reorganize test_dumm.py
======================================================================== test session starts ========================================================================
platform linux2 -- Python 2.7.12, pytest-3.10.1, py-1.5.4, pluggy-0.7.1 -- /tmp/tmp.w95BqE188N/bin/python2
cachedir: .pytest_cache
Using --randomly-seed=1566829391
rootdir: /home/rkalaiselvan, inifile: pytest.ini
plugins: randomly-1.2.3, timeout-1.3.1, cov-2.6.0, mock-1.10.0, ordering-0.6
collected 4 items

test_dumm.py::test_three PASSED
test_dumm.py::test_four PASSED
test_dumm.py::test_two PASSED
test_dumm.py::test_one PASSED

(tmp.w95BqE188N) rkalaiselvan@dev-rkalaiselvan:~/$ py.test -p no:randomly test_dumm.py
======================================================================== test session starts ========================================================================
platform linux2 -- Python 2.7.12, pytest-3.10.1, py-1.5.4, pluggy-0.7.1 -- /tmp/tmp.w95BqE188N/bin/python2
cachedir: .pytest_cache
Using --randomly-seed=1566829391
rootdir: /home/rkalaiselvan, inifile: pytest.ini
plugins: randomly-1.2.3, timeout-1.3.1, cov-2.6.0, mock-1.10.0, ordering-0.6
collected 4 items

test_dumm.py::test_three PASSED
test_dumm.py::test_four PASSED
test_dumm.py::test_two PASSED
test_dumm.py::test_one PASSED

Answer 8

Pytest by default it executes the steps in randomaly. So use this option for executing in order.

Install with:

pip install pytest-order

Write the test method as like below:

import pytest

@pytest.mark.order(1)
def test_foo():
    assert True

@pytest.mark.order(2)
def test_bar():
    assert True

Answer 9

Make Sure you have installed pytest-ordering package. To confirm go to Pycharm settings>>Project Interpreter >> and look for pytest-ordering : If it is not available install it. Pycharm settings>>Project Interpreter >> Click on + button and search pytest-ordering install it. Voila!! It will definitely work.