Jenkins Not Outputting Junit Report Info From File

Question

Problem

A junit-formatted report is not being picked up by Jenkins, causing the reports not to be listed in the project's status screen.

Details

The junit-formatted report data is generated by a testing framework called Karma-runner (formerly Testacular). The file being ignored is created in /target/surefire-reports -- the same location as where surefire-generated reports are created. The report data looks practically the same as that generated by the maven surefire plugin except that its parent element is <testsuites> instead of <testsuite> -- <testsuite> is what the surefire generated reports have as the parent element of a report file. Here's a snippet from the karma-generated junit-formatted report, named TEST-karma.resultsTest.xml:

Junit-formatted Karma-generated report file, TEST-karma.resultsTest.xml

<?xml version="1.0"?>
<testsuites>
  <testsuite name="PhantomJS 1.9 (Mac)" package="karma.tests" timestamp="2013-04-10T13:32:26" id="0" hostname="jgmbp.local" tests="16" errors="0" failures="0" time="0.069">
    <properties>
      <property name="browser.fullName" value="Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.0 Safari/534.34"/>
    </properties>
    <testcase name="successfully cancels a new member" time="0.006" classname="karma.tests PhantomJS 1.9 (Mac).Household Controller"/>
    <testcase name="should parse an existing re-eval create date, setting the data in scope" time="0.003" classname="karma.tests PhantomJS 1.9 (Mac).Re-Eval Controller"/>
    <system-out><![CDATA[
]]></system-out>
    <system-err/>
  </testsuite>
</testsuites>

The Karma tests are run during the test phase of my maven build. I've tried creating projects that only produce this one junit-report file as well as running the build so that all junit tests and karma tests generate report files. Jenkins will always pick up the surefire tests, but never the karma tests.

Thanks for any input!

Answer 1

You can publish your Karma test results in Jenkins even when building a Maven project, if you fool Jenkins with a little hack: run the Surefire plugin in also in the Javascript module you run the Karma tests in.

The surefire plugin will not find any JUnit tests in your Javascript module, but it will notify Jenkins that Surefire test results are available.

Below is a sample pom.xml, and snippets from Gruntfile.js, which runs Karma tests and a JSHint code analysis, and causes Jenkins to publish the test results and the code analysis results.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.xxx.yyyy</groupId>
<artifactId>zzzzz</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Zzzzz</name>

<build>
    <plugins>
        <plugin>
            <artifactId>maven-antrun-plugin</artifactId>
            <version>1.7</version>
            <executions>
                <execution>
                    <phase>generate-resources</phase>
                    <configuration>
                        <target>
                            <exec
                                dir="${basedir}"
                                executable="npm"
                                failonerror="true">
                                <arg value="install"/>
                            </exec>
                            <exec
                                dir="${basedir}"
                                executable="bower"
                                failonerror="true">
                                <arg value="install"/>
                            </exec>
                            <exec
                                    dir="${basedir}"
                                    executable="grunt"
                                    failonerror="true">
                                <arg value="--no-color"/>
                                <arg value="build"/>
                            </exec>
                        </target>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>

                <execution>
                    <id>jshint</id>
                    <phase>test</phase>
                    <configuration>
                        <target>
                            <exec
                                dir="${basedir}"
                                executable="grunt"
                                failonerror="false">
                                <arg value="--no-color"/>
                                <arg value="karma:run"/>
                            </exec>
                            <exec
                                dir="${basedir}"
                                executable="grunt"
                                failonerror="false">
                                <arg value="--no-color"/>
                                <arg value="jshint:jenkins"/>
                            </exec>
                        </target>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>

            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.12.4</version>
            <executions>
                <!-- This is a hack to get Jenkins to publish Karma test results when running a Maven project: we run 0 surefire tests, so Jenkins publishes the report of the Karma tests. -->
                <execution>
                    <id>dummySureFire</id>
                    <phase>test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <version>2.12</version>
            <executions>
                <!-- This is a hack to get Jenkins to publish JSHint results when running a Maven project: we run checkstyle, so Jenkins publishes the report of the JSHint run. -->
                <execution>
                    <id>dummyCheckStyle</id>
                    <phase>test</phase>
                    <goals>
                        <goal>checkstyle</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
<reporting>
    <plugins>
        <plugin>
            <artifactId>maven-surefire-report-plugin</artifactId>
            <version>2.12.4</version>
        </plugin>
    </plugins>
</reporting>
</project>

In Gruntfile.js we have the following. Outputfile name must begin with TEST- in order for it to get published:

karma: {
        run: { // produces reports for Jenkins
            configFile: 'karma.conf.js',
            singleRun: true,
            reporters : ['junit', 'coverage'],
            junitReporter : {
                outputFile: 'target/surefire-reports/TEST-results.xml'
            },
        ...

And for JSHint in Gruntfile.js:

jshint: {
        all: [
            'Gruntfile.js',
            '<%= yeoman.app %>/scripts/{,*/}*.js'
        ],
        options: {
            jshintrc: '.jshintrc',
            reporter: require('jshint-stylish')
        },
        jenkins: {
            options: {
                reporter: 'checkstyle',
                reporterOutput: "target/checkstyle-result.xml"
            },
            src: [
                'Gruntfile.js',
                '<%= yeoman.app %>/scripts/{,*/}*.js'
            ]
        }
    }

In your Jenkins job configuration you just need to check the "Publish Checkstyle analysis results" box in "Build Settings" section in order to have Jenkins publish the JSHint results. No additional Jenkins job configuration is necessary for publishing the test results.

Answer 2

Finally had the time to figure it out, thanks to amey and Wouter's input.

The key is that I'm building a maven project, and that won't work with displaying the karma tests. The only way to display karma results is to add a post build report for Junit tests. For some reason, maven-based jobs do not give you the option to add a junit report.

The solution is to create a free form job instead of a maven job. You can configure the free form job to build the maven project just fine. As long as you add the publish junit task per amey's post above, all is well.

Answer 3

Although this question is quite old:

There is a solution that works also with a Maven project in Jenkins. Problem is that the Jenkins recorder recording the tests runs directly after the surefire:test goal. So in order for your tests to get picked up, whichever plugin you use to run karma must run BEFORE the test phase, for example in the process-test-classes phase (because you cannot put something in the same phase before an existing plugin).

Answer 4

1) Have you entered the path to the report where Jenkins should find the Junit report? This can be done in the Post build action - Publish JUnit test result report

2) Or you could use the Performance Plugin to pass your Junit results. It again is a post build action. Select that plugin in the configuration of that Jenkins job.

Add a new report - Junit and enter the path to the report target/surefire-reports/*.xml (Note - I have removed the first slash as the path is with respect to the current workspace).

Answer 5

I've had the same problem. The fact that test results are wrapped in a single <testsuites/> tag is not the problem. Jenkins' maven project plugin is the problem. The typical maven project usues this because it makes configuring your build easier, we did this. I embedded Karma in my pom.xml to be able to build everything at once, using the maven-exec-plugin like so:

        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>1.2.1</version>
            <executions>
                <execution>
                    <id>jsunit</id>
                    <phase>test</phase>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <successCodes>
                    <successCode>0</successCode>
                    <successCode>1</successCode><!-- on failing test, don't get a build error -->
                </successCodes>
                <executable>${basedir}/src/test/javascript/test.bat</executable>
                <workingDirectory>${basedir}/src/test/javascript</workingDirectory>
            </configuration>
        </plugin>

The maven jenkins plugin indeed refuses to pick up the generated XML. If you, however, configure a "basic" jenkins job instead of a maven2 specific one using the plugin, it will work. Add a post-step, "publish JUNIT reports" and use target/surefire-reports/*.xml. As a normal build step, you can still add the "execute maven task" step.

It's still not perfect as local builds using mvn test seem to ignore the XML (maven-surefire-plugin simply generates the XML and stops the build if a JUnit test fails). You can stop the build if a javascript test fails by using the success codes in the exec-maven-plugin but the error message still is not clear...

Answer 6

I'm late in this game, but I had the same problem and I stumbled upon this year-old post.

Anyhow, I found, in my opinion, a more elegant solution by leveraging maven-karma-plugin. This allows me to continue using Maven Project jobs instead of switching them to Freestyle Project jobs in Jenkins.

The detailed solution is posted at http://myshittycode.com/2014/10/23/jenkins-getting-karma-generated-test-results-to-appear-in-maven-project-job/ in case anyone gets stuck with this same problem.

Thanks.