Jenkins, how to check regressions against another job

Question

When you set up a Jenkins job various test result plugins will show regressions if the latest build is worse than the previous one.

We have many jobs for many projects on our Jenkins and we wanted to avoid having a 'job per branch' set up. So currently we are using a parameterized build to build eg different development branches using a single job.

But that means when I build a new branch any regressions are measured against the previous build, which may be for a different branch. What I really want is to measure regressions in a feature branch against the latest build of the master branch.

I thought we should probably set up a separate 'master' build alongside the parameterized 'branches' build. But I still can't see how I would compare results between jobs. Is there any plugin that can help?

UPDATE

I have started experimenting in the Script Console to see if I could write a post-build script... I have managed to get the latest build of master branch in my parameterized job... I can't work out how to get to the test results from the build object though.

The data I need is available in JSON at
http://<jenkins server>/job/<job name>/<build number>/testReport/api/json?pretty=true

...if I could just get at this data structure it would be great!

I tried using JsonSlurper to load the json via HTTP but I get 403, I guess because my script has no auth session.

I guess I could load the xml test results from disk and parse them in my script, it just seems a bit stupid when Jenkins has already done this.

Anentropic Anentropic · Accepted Answer · 2015-11-03T13:20:04

I eventually managed to achieve everything I wanted, using a Groovy script in the Groovy Postbuild Plugin

I did a lot of exploring using the script console http://<jenkins>/script and also the Jenkins API class docs are handy.

Everyone's use is going to be a bit different as you have to dig down into the build plugins to get the info you need, but here's some bits of my code which may help.

First get the build you want:

def getProject(projectName) {
    // in a postbuild action use `manager.hudson`
    // in the script web console use `Jenkins.instance`
    def project = manager.hudson.getItemByFullName(projectName)
    if (!project) {
        throw new RuntimeException("Project not found: $projectName")
    }
    project
}

// CloudBees folder plugin is supported, you can use natural paths:
project = getProject('MyFolder/TestJob')

build = project.getLastCompletedBuild()

The main test results (jUnit etc) seem to be available directly on the build as:

result = build.getTestResultAction()
// eg
failedTestNames = result.getFailedTests().collect{ test ->
    test.getFullName()
}

To get the more specialised results from eg Violations plugin or Cobertura code coverage you have to look for a specific build action.

// have a look what's available:
build.getActions()

You'll see a list of stuff like:

[hudson.plugins.git.GitTagAction@2b4b8a1c,
 hudson.scm.SCMRevisionState$None@40d6dce2,
 hudson.tasks.junit.TestResultAction@39c99826,
 jenkins.plugins.show_build_parameters.ShowParametersBuildAction@4291d1a5]

These are instances, the part in front of the @ sign is the class name so I used that to make this method for getting a specific action:

def final VIOLATIONS_ACTION = hudson.plugins.violations.ViolationsBuildAction
def final COVERAGE_ACTION = hudson.plugins.cobertura.CoberturaBuildAction

def getAction(build, actionCls) {
    def action = build.getActions().findResult { act ->
        actionCls.isInstance(act) ? act : null
    }
    if (!action) {
        throw new RuntimeException("Action not found in ${build.getFullDisplayName()}: ${actionCls.getSimpleName()}")
    }
    action
}

violations = getAction(build, VIOLATIONS_ACTION)

// you have to explore a bit more to find what you're interested in:
pylint_count = violations?.getReport()?.getViolations()?."pylint"

coverage = getAction(build, COVERAGE_ACTION)?.getResults()
// if you println it looks like a map but it's really an Enum of Ratio objects
// convert to something nicer to work with:
coverage_map = coverage.collectEntries { key, val -> [key.name(), val.getPercentageFloat()] }

With these building blocks I was able to put together a post-build script which compared the results for two 'unrelated' build jobs, then using the Groovy Postbuild plugin's helper methods to set the build status.

Hope this helps someone else.

Jenkins, how to check regressions against another job

1 Answers