Jenkins set build failed if number of tests decreases?

Question

Is there a way to mark a test run failed if the number of tests decreases in jenkins. We have a test report in JUnit. For some reasons, sometimes the number of tests decreases. Often that are critical errors. Is there a way to say jenkins that in such case the test status should be red? (Maybe some plugin)

Thanks a lot for any hint!

Answer 1

There is a way.

Install the Groovy Postbuild Plugin and add a Post Build Step

Post-Build-Actions -> Add Step -> Groovy Postbuild

import jenkins.model.*

def currentBuild = manager.build
def totalCurrent 
def totalPrevious

// evaluate test count of current Build
def result = currentBuild .getAction(hudson.tasks.junit.TestResultAction.class).result
if (result != null) {
  totalCurrent = result.hasProperty( 'totalCount' ) ? result.totalCount : null
} 

// evaluate test count of previous Build
result = currentBuild .previousBuild.getAction(hudson.tasks.junit.TestResultAction.class).result
if (result != null) {
  totalPrevious = result.hasProperty( 'totalCount' ) ? result.totalCount : null
} 

// fail the build if test count reduced
if(totalCurrent < totalPrevious) {
  manager.buildFailure()
}

You'll may need to add some more nullsafe checks.

Answer 2

Even if there is a way to do this, it sounds like a bad idea considering the fact that your number of tests executed could actually decrease from build to build for legitimate reasons.

First I would try to find out why this is happening unexpectedly.