python - Is there a way to specify which pytest tests to run from a file?

Question

Is there a way to select pytest tests to run from a file? For example, a file foo.txt containing a list of tests to be executed:

tests_directory/foo.py::test_001
tests_directory/bar.py::test_some_other_test

Or, is there a way to select multiple tests, having no common pattern in test name, from different directories with pytest?

pytest -k <pattern> allows a single pattern.

One option is to have a pytest.mark against each test, but my requirement is to run different combination of tests from different files.

Is there a way to specify multiple patterns and a test file name for each pattern?

Or

Is there a way to specify the exact test paths in a file and feed that file as an input to pytest?

Or

Is there a hook function that can be utilized for this purpose?

Answer 1

You can use -k option to run test cases with different patterns:

py.test tests_directory/foo.py tests_directory/bar.py -k 'test_001 or test_some_other_test'

This will run test cases with name test_001 and test_some_other_test deselecting the rest of the test cases.

Note: This will select any test case starting with test_001 or test_some_other_test. For example, if you have test case test_0012 it will also be selected.

Answer 2

Specifying tests / selecting tests

Pytest supports several ways to run and select tests from the command-line.

Run tests in a module

pytest test_mod.py

Run tests in a directory

pytest testing/

Run tests by keyword expressions

pytest -k "MyClass and not method"

This will run tests which contain names that match the given string expression, which can include Python operators that use filenames, class names and function names as variables. The example above will run TestMyClass.test_something but not TestMyClass.test_method_simple.

Run tests by node ids

Each collected test is assigned a unique nodeid which consist of the module filename followed by specifiers like class names, function names and parameters from parametrization, separated by :: characters.

To run a specific test within a module:

pytest test_mod.py::test_func

Another example specifying a test method in the command line:

pytest test_mod.py::TestClass::test_method

Run tests by marker expressions

pytest -m slow

Will run all tests which are decorated with the @pytest.mark.slow decorator.

For more information see marks.

Run tests from packages

pytest --pyargs pkg.testing

This will import pkg.testing and use its filesystem location to find and run tests from.

Source: https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests

Answer 3

My answer provides a ways to run a subset of test in different scenarios.

Run all tests in a project

pytest

Run tests in a Single Directory

To run all the tests from one directory, use the directory as a parameter to pytest:

pytest tests/my-directory

Run tests in a Single Test File/Module

To run a file full of tests, list the file with the relative path as a parameter to pytest:

pytest tests/my-directory/test_demo.py

Run a Single Test Function

To run a single test function, add :: and the test function name:

pytest -v tests/my-directory/test_demo.py::test_specific_function

-v is used so you can see which function was run.

Run a Single Test Class

To run just a class, do like we did with functions and add ::, then the class name to the file parameter:

pytest -v tests/my-directory/test_demo.py::TestClassName

Run a Single Test Method of a Test Class

If you don't want to run all of a test class, just one method, just add another :: and the method name:

pytest -v tests/my-directory/test_demo.py::TestClassName::test_specific_method

Run a Set of Tests Based on Test Name

The -k option enables you to pass in an expression to run tests that have certain names specified by the expression as a substring of the test name. It is possible to use and, or, and not to create complex expressions.

For example, to run all of the functions that have _raises in their name:

pytest -v -k _raises

Answer 4

If you have the same method name in two different classes and you just want to run one of them, this works:

pytest tests.py -k 'TestClassName and test_method_name'

Answer 5

Method 1: Randomly selected tests. Long and ugly.

python -m pytest test/stress/test_performance.py::TestPerformance::test_continuous_trigger test/integration/test_config.py::TestConfig::test_valid_config

Method 2: Use Keyword Expressions.

Note: I am searching by testcase names. Same is applicable to TestClass names.

Case 1: The below will run whichever is found. Since we have used 'OR' .

python -m pytest -k 'test_password_valid or test_no_configuration'

Lets say the two above are actually correct, 2 tests will be run.

Case 2: Now an incorrect name and another correct name.

python -m pytest -k 'test_password_validzzzzzz or test_no_configuration' 

Only one is found and run.

Case 3: If you want all tests to run or no one, then use AND

python -m pytest -k 'test_password_valid and test_no_configuration'

Both will be run if correct or none.

Case 4: Run test only in one folder:

python -m pytest test/project1/integration -k 'test_password_valid or test_no_configuration'

Case 5: Run test from only one file.

python -m pytest test/integration/test_authentication.py -k 'test_password_expiry or test_incorrect_password'

Case 6: Run all tests except the match.

python -m pytest test/integration/test_authentication.py -k 'not  test_incorrect_password'

Answer 6

Maybe using pytest_collect_file() hook you can parse the content of a .txt o .yaml file where the tests are specify as you want, and return them to the pytest core.

A nice example is shown in the pytest documentation. I think what you are looking for.

Answer 7

According to the doc about Run tests by node ids

since you have all node ids in foo.txt, just run

pytest `cat foo.txt | tr '\n' ' '`

this is same with below command (with file content in the question)

pytest tests_directory/foo.py::test_001 tests_directory/bar.py::test_some_other_test

Answer 8

Here's a possible partial answer, because it only allows selecting the test scripts, not individual tests within those scripts.

And it also limited by my using legacy compatibility mode vs unittest scripts, so not guaranteeing it would work with native pytest.

Here goes:

1. create a new dictory, say subset_tests_directory.
2. ln -s tests_directory/foo.py

3. ln -s tests_directory/bar.py

4. be careful about imports which implicitly assume files are in test_directory. I had to fix several of those by running python foo.py, from within subset_tests_directory and correcting as needed.

5. Once the test scripts execute correctly, just cd subset_tests_directory and pytest there. Pytest will only pick up the scripts it sees.

Another possibility is symlinking within your current test directory, say as ln -s foo.py subset_foo.py then pytest subset*.py. That would avoid needing to adjust your imports, but it would clutter things up until you removed the symlinks. Worked for me as well.