Why does pytest-xdist make my tests run slower, not faster?

16
votes

I'm porting a ~2000 method test suite from nose to pytest because django-nose didn't support parallelization well. Swapping out nose for pytest seemed to work pretty well, and after adding python_files to pytest.ini it found almost all of our tests.

The big downside is that when I run with -n 4, the test suite becomes slower than without the -n flag at all. Running on a ~10% subset of the entire suite, it appears to be about 20-30% flat slowdown, though the timings I've taken are rather noisy. This makes some sense as overhead, but no matter how many processes I select, the timing never goes down.

Running with --durations=20 shows each setup phase takes a few seconds longer per process, and each other test gets marginally slower.

With -vvv to list the tests as they're run, the output is almost entirely serialized:

api/tests/VERSION_NUMBER.py::ATestCase::test_forbidden_methods <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_access_token <- api/testcases.py
api/tests/VERSION_NUMBER.py::ATestCase::test_create <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_create <- api/testcases.py
api/tests/VERSION_NUMBER.py::ATestCase::test_delete <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_delete <- api/testcases.py
api/tests/VERSION_NUMBER.py::ATestCase::test_patch <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_patch <- api/testcases.py
api/tests/VERSION_NUMBER.py::ATestCase::test_put <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_put <- api/testcases.py
api/tests/VERSION_NUMBER.py::ATestCase::test_retrieve <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_retrieve <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_access_token <- api/testcases.py
[gw0] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_forbidden_methods <- api/testcases.py
api/tests/VERSION_NUMBER.py::ATestCase::test_list <- api/testcases.py
[gw0] PASSED api/tests/VERSION_NUMBER.py::ATestCase::test_list <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_delete <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_access_token <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_create <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_create <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_list <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_list <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_patch <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_patch <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_put <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_put <- api/testcases.py
[gw0] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_delete <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_forbidden_methods <- api/testcases.py
api/tests/VERSION_NUMBER.py::BTestCase::test_retrieve <- api/testcases.py
[gw0] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_forbidden_methods <- api/testcases.py
[gw1] PASSED api/tests/VERSION_NUMBER.py::BTestCase::test_retrieve <- api/testcases.py

With few exceptions, it's almost always "start a test, get PASSED from a worker" for the entire log. This leads me to believe that something is serializing the tests, but I'm baffled as to what.

I've tried disabling all pytest plugins except for pytest itself, pytest-xdist, and pytest-django, with no change.

3
pytestpytest-django
I am seeing similar problems. Were you able to get to the bottom of this issue in your case? – JimmidyJoo
Run the tests with --trace-config flag. It will show the set up of fixtures. xdist speeds up the test execution only when tests are independent. By the looks of it, it seems like your tests depend on the execution of other. If you are sharing data or objects between tests, have a look at them. Changing the scope of user defined fixtures should give you more insight too. – SilentGuy

3 Answers

1
votes

Read https://github.com/pytest-dev/pytest-xdist/blob/master/OVERVIEW.md and you will guess why it can be considerably slower in particular cases.

When parallelized may be slower:

  • total test duration is quite slow (under 2minutes) - bootraping pytest workers adds extra time, if that is bigger than the benefit...
  • your tests already hog a shared finite resource like disk or networking, so running in parallel may make it slower
0
votes

Make sure you do correctly control the way tests parallelism are used, follow the answer on Is there a way to control how pytest-xdist runs tests in parallel? to verify you do utilize parameters correctly, especially pay attention to the way dist parameter is set. I would suggest to set it --dist=loadfile.

0
votes

If your tests are not already pretty long, it is easy to endup with slower runtime using xdist than using serial, that is known and documented on https://github.com/pytest-dev/pytest-xdist/issues/346

AFAIK, there is no clear solution for it and it should be up to you to decide which project benefits from xdist or not, or to tune the number of workers in order to optimize the outcomes.