How to not mark Jenkins job as FAILURE when pytest tests fail

Question

I have a Jenkins setup with a pipeline that uses pytest to run some test suites. Sometimes a test fails and sometimes the test environment crashes (random HTTP timeout, external lib error, etc.). The job parses the XML test result but the build is marked as FAILURE as long as pytest returns non-zero.

I want Jenkins to get exit code zero from pytest even if there are failed tests but I also want other errors to be marked as failures. Are there any pytest options that can fix this? I found pytest-custom_exit_code but it can only suppress the empty test suite error. Maybe some Jenkins option or bash snippet?

A simplified version of my groovy pipeline:

pipeline {
    stages {
        stage ('Building application') {
            steps {
                sh "./build.sh"
            }
        }
        stage ('Testing application') {
            steps {
                print('Running pytest')
                sh "cd tests && python -m pytest"
            }
            post {
                always {
                    archiveArtifacts artifacts: 'tests/output/'
                    junit 'tests/output/report.xml'
                }
            }
        }
    }
}

I have tried to catch exit code 1 (meaning some tests failed) but Jenkins still received exit code 1 and marks the build as FAILURE:

sh "cd tests && (python -m pytest; rc=\$?; if [ \$rc -eq 1 ]; then exit 0; else exit \$rc; fi)"

Can you prevent any code? It is possible to set the result to unstable but how is of course depending on your implementation. — Michael Kemmerzell
As I understand, pyest return non zero for any error. Do you want to get the exit code and if is different to zero, mark as unstable? — JRichardsz
As I understand, the default Jenkins behavior is to mark a build as UNSTABLE if the test result is not successful and FAILURE if the build failed to produce a test result. I want to do this to be able to tell if it was a test failure or an environment error. — Marcus Ahlberg
Possible duplicate of How to mark a build unstable in Jenkins when running shell scripts — Joao Vitorino
Not really. My problem is that the job is marked as FAILURE. I have changed the title to better describe the problem. — Marcus Ahlberg

Fabian Ritzmann Fabian Ritzmann · Accepted Answer · 2019-10-09T11:14:24

Your attempt does not work because Jenkins runs the shell with the errexit (-e) option enabled and that causes the shell to exit right after the pytest command before it reaches your if statement. There is a one-liner however that will work because it is executed as one statement: https://stackoverflow.com/a/31114992/1070890

So your build step would look like this:

sh 'cd tests && python -m pytest || [[ $? -eq 1 ]]'

How to not mark Jenkins job as FAILURE when pytest tests fail

2 Answers