How to disable skipping a test in pytest without modifying the code?

8
votes

I have inherited some code that implements pytest.mark.skipif for a few tests. Reading through the pytest docs, I am aware that I can add conditions, possibly check for environment variables, or use more advanced features of pytest.mark to control groups of tests together. Unfortunately nothing in the docs so far seems to solve my problem.

I'm looking to simply turn off any test skipping, but without modifying any source code of the tests. I just want to run pytest in a mode where it does not honor any indicators for test skipping. Does such a solution exist with pytest?

5
pythonunit-testingpytestpython-decorators
maybe write @pytest.mark.skip above the test case, for pytest to skipp that test case - sahasrara62
The tests already have this. I am asking about how to disable that. I'm not asking how to disable or skip the test itself. I'm asking how to turn off skipping, so that no test can be skipped at all. It is for diagnostic purposes to examine why tests that are not skipped in a separate environment are failing. Running them locally is very hard because of the skipif annotations, but this type of debugging is rare so I want the skipif annotations to remain, and there is too much code to go and modify all of the places where skipif is used. I'd also strongly prefer not to monkeypatch. - ely
Doing a global find and replace in your IDE shouldn’t be terribly difficult. Replace “skipif” with some word like “temp_enable” it should work. Just put it back when you are done. - soundstripe
What's the condition for skipif? - gmds
@soundstripe I'd like this to be configurable, so that in the future if this type of debugging issue happens again, I can just easily re-run with no skipping. Needing to find/replace each time should be avoided if possible. - ely

5 Answers

3
votes

Create a conftest.py with the following contents:

import pytest
import _pytest.skipping


def pytest_addoption(parser):
    parser.addoption(
        "--no-skips",
        action="store_true",
        default=False, help="disable skip marks")


@pytest.hookimpl(tryfirst=True)
def pytest_cmdline_preparse(config, args):
    if "--no-skips" not in args:
        return

    def no_skip(*args, **kwargs):
        return

    _pytest.skipping.skip = no_skip

the use --no-skip in command line to run all testcases even if some testcases with pytest.mark.skip decorator

2
votes

Ok the implementation does not allow for this with zero modifications. You’ll need a custom marker. Add the following to your conftest.py then change all skipif marks to custom_skipif. Use pytest --no-skips. 

import pytest
from _pytest.mark.evaluate import MarkEvaluator

def pytest_addoption(parser):
    parser.addoption(
        "--no-skips", action="store_true", default=False, help="disable custom_skip marks"
    )

@hookimpl(tryfirst=True)
def pytest_runtest_setup(item):
    if item.config.getoption('--no-skips'):
        return

    # Check if skip or skipif are specified as pytest marks
    item._skipped_by_mark = False
    eval_skipif = MarkEvaluator(item, "custom_skipif")
    if eval_skipif.istrue():
        item._skipped_by_mark = True
        pytest.skip(eval_skipif.getexplanation())

    for skip_info in item.iter_markers(name="custom_skip"):
        item._skipped_by_mark = True
        if "reason" in skip_info.kwargs:
            pytest.skip(skip_info.kwargs["reason"])
        elif skip_info.args:
            pytest.skip(skip_info.args[0])
        else:
            pytest.skip("unconditional skip")

    item._evalxfail = MarkEvaluator(item, "xfail")
    check_xfail_no_run(item)

The implementation is copied and modified from pytest itself in skipping.py.

1
votes

An easy workaround is to monkeypatch pytest.mark.skipif in your conftest.py:

import pytest

old_skipif = pytest.mark.skipif

def custom_skipif(*args, **kwargs):
    return old_skipif(False, reason='disabling skipif')

pytest.mark.skipif = custom_skipif
1
votes

A workaround to ignore skip marks is to remove them programmatically. Create a conftest.py with the following contents:

def pytest_collection_modifyitems(items):
    for item in items:
        for node in reversed(item.listchain()):
            node.own_markers = [m for m in node.own_markers if m.name not in ('skip', 'skipif')]

However, this messes with pytest internals and can easily break on pytest updates; the proper way of ignoring skips should be defining your custom skipping mechanism, for example:

@pytest.hookimpl(tryfirst=True)
def pytest_runtest_setup(item):
    mark = item.get_closest_marker(name='myskip')
    if mark:
        condition = next(iter(mark.args), True)
        reason = mark.kwargs.get('reason', 'custom skipping mechanism')
        item.add_marker(pytest.mark.skipif(not os.getenv('PYTEST_RUN_FORCE_SKIPS', False) and condition, reason=reason), append=False)

Annotate the tests with @pytest.mark.myskip instead of @pytest.mark.skip and @pytest.mark.myskip(condition, reason) instead of @pytest.mark.skipif(condition, reason):

@pytest.mark.myskip
def test_skip():
    assert True


@pytest.mark.myskip(1 == 1, reason='my skip')
def test_skipif():
    assert True

On a regular run, myskip will behave same way as pytest.mark.skip/pytest.mark.skipif. Setting PYTEST_RUN_FORCE_SKIPS will disable it:

$ PYTEST_RUN_FORCE_SKIPS=1 pytest -v
...
test_spam.py::test_skip PASSED
test_spam.py::test_skipif PASSED
...

Of course, you shouldn't use pytest.mark.skip/pytest.mark.skipif anymore as they are won't be influenced by the PYTEST_RUN_FORCE_SKIPS env var.

0
votes

Now pytest_cmdline_preparse is deprecated, they suggest to move on pytest_load_initial_conftests info.

Someone have any idea to how move further?