How can I get, on a per test case, all robotframework warnings in the email-ext groovy script?

2
votes

Background

Here is a background on the configuration

  • I have tests written in Robot Framework
  • I am using Jenkins and installed the robotframework and email-ext plugin
    • I have a Jenkins job scheduled to run periodically
      • The jenkins jobs runs robot tests
      • Then sends an email (using email-ext) with the test results
      • I am using a groovy script to generate the email content.

The groovy script that generates the email content is using the robot APIs to get data (e.g.: total tests, total failed, pass percentage) and information about all failed test cases.

For all failed test cases, a list of RoboCaseResult, I am including:

  • Test criticality
  • Parent suite
  • Test Name
  • Error message

This is currently working. Here's my Groovy Script achieves this:

def actionslist = build.actions // List<hudson.model.Action>
def doRobotResultsExist = false
actionslist.each() { action ->
    if( action.class.simpleName.equals("RobotBuildAction") ) { // hudson.plugins.robot.RobotBuildAction
        doRobotResultsExist = true
        displaycritical = (action.getOverallPassPercentage() != action.getCriticalPassPercentage())
        %>
        <h3>RobotFramework Results</h3>
        <table>
            <tr>
                <td>Detailed Report:</td>
                <td><a href="${rooturl}${build.url}robot/report/<%= action.getLogHtmlLink() %>" target="_new"><%= action.getLogHtmlLink() %></a></td>
            </tr>
            <!--
            <tr>
                <td>Pass Percentage:</td>
                <td><%= action.overallPassPercentage %>%</td>
            </tr>
            -->
            <tr>
                <td>Overall Pass Ratio:</td>
                <td><%= action.getTotalCount() - action.getFailCount() %>/<%= action.getTotalCount() %></td>
            </tr>
            <tr>
                <td>Pass Percentage:</td>
                <td><%= action.getOverallPassPercentage() %>%</td>
            </tr>
            <% 
            if (displaycritical) {
            %>
            <tr>
                <td>Critical Pass Percentage:</td>
                <td><%= action.getCriticalPassPercentage() %>%</td>
            </tr>
            <% } //if displaycrital %>
            
            <tr>
                <td>Total Failed:</td>
                <td><%= action.getFailCount() %></td>
            </tr>
        </table>
        <%
        //action.result returns  hudson.plugins.robot.model.RobotResult
        //action.retult.getAllFailedCases() returns a list of hudson.plugins.robot.model.RobotCaseResult
        def allFailedTests = action.result.getAllFailedCases() // hudson.plugins.robot.model.RobotCaseResult
        
        if (!allFailedTests.isEmpty()) {
            i = 0
            %>
            <table cellspacing='0' cellpadding='1' border='1'>
            <tr class='bg1'>
                <% if (displaycritical) { %><th>Tagged Critical</th><% } //if displaycrital %>
                <th>Suite</th>
                <th>Failed Test Case</th>
                <th>Error message</th>
            </tr>
            <%
            //allFailedTests.each() { testresult ->
            //    def testCaseResult = testresult
            allFailedTests.each() { testCaseResult ->
                i++
                print "<tr " + ( (i % 2) == 0 ? "class='bg2'" : "") + " >"
                if (displaycritical) {
                    print "<td>" + (testCaseResult.isCritical()? "<font color='red'><b>YES</b></font>": "no" )+ "</td>"
                }
                print "<td>" + testCaseResult.getParent().getRelativePackageName(testCaseResult.getParent()) + "</td>"
                print "<td>" + testCaseResult.getDisplayName() + "</td>"
                print "<td>" + testCaseResult.getErrorMsg() + "</td>"
                print "</tr>"
            } // for each failed test
            %>
            </table>
            <%
        } // if list of failed test cases is not empty
    } // end action is RobotBuildAction
} // end of each actions

This generated something like this

=========================================================================
| Tagged Critical | Suite  | Failed Test Case | Error Message           |
=========================================================================
|  YES            | Fruits | Get Apples       | Could not get apples    |
+-----------------+--------+------------------+-------------------------+
|  NO             | Fruits | Eat Apple        | Could not find an apple |
=========================================================================

The issue

For each failed test case, I want to include all WARNING that were raised. But I'm not able to find an API for this, so I opened an enhancement ticket against the Jenkins plugin. The Jenkins robotframework plugin maintainer may not reply to my request in the timeframe that I need a solution.

How can I include, via the groovy script, all the warnings that were raised during a robot test? That is, I want to get the following

================================================================================================
| Tagged Critical | Suite  | Failed Test Case | Error Message           | Warnings             |
================================================================================================
|  YES            | Fruits | Get Apples       | Could not get apples    | No baskets available |
+-----------------+--------+------------------+-------------------------+----------------------+
|  NO             | Fruits | Eat Apple        | Could not find an apple |                      |
================================================================================================
1
groovyjenkinsjenkins-pluginsrobotframeworkemail-ext

1 Answers

3
votes

I'm not familiar with the robot plugin API, so I can't say whether the information you want is available to you or not from there. What I do know is that the information is available in the output.xml file that is generated by robot. This file is very easy to parse. Here's the top part of a file generated by a test with one log keyword:

<robot generated="20140527 19:46:02.095" generator="Robot 2.8.1 (Python 2.7.2 on darwin)">
  <suite source="/tmp/example.robot" id="s1" name="Example">
    <test id="s1-t1" name="Example of a warning">
      <kw type="kw" name="BuiltIn.Log">
        <doc>Logs the given message with the given level.</doc>
          <arguments>
            <arg>This is a warning</arg>
            <arg>warn</arg>
          </arguments>
          ...

Another solution would be to create a custom listener that listens for warning and error message, saves them, and then creates your email message on the fly as the test is running. Every time you get an end_test message you can print any errors or warnings that you've detected to that point. Once the suite is finished running you'll have your email message ready to send.

The following is a quick example, though it doesn't give the exact output you want (I'm too lazy to calculate the maximum width of each column):

class exampleListener():
    ROBOT_LISTENER_API_VERSION = 2

    def __init__(self, filename="messages.txt"):
        self.outfile = open(filename, "w")
        self.outfile.write("Errors and Warnings\n")
        self.current_test = None
        self.current_messages = []
        self.current_suite = None

    def start_suite(self, name, attrs):
        self.current_suite = name

    def start_test(self, name, attrs):
        self.current_test = name
        self.current_messages = []

    def log_message(self, data):
        self.current_messages.append((data["level"], data["message"]))

    def end_test(self, name, attrs):
        for (type, text) in self.current_messages:
            if type == "ERROR":
                self.outfile.write("| %s | %s | %s\n" % (self.current_suite, self.current_test, text))
            elif type == "WARN":
                self.outfile.write("| %s | %s | | %s\n" % (self.current_suite, self.current_test, text))

        self.current_messages = []
        self.current_test = None

    def end_suite(self, name, attrs):
        self.outfile.close()

You would use this by using the --listener argument, for example:

pybot --listener ExampleListener.py my_suite