Determine Failed Stage in Jenkins Declarative Pipeline

Question

How do I report the stage in which a declarative pipeline failed? In the fail block, I want to get failedStage.name and report it (eventually to slack).

pipeline {
    agent { label 'master'}
    stages {
        stage('Ok') {
            steps {
                echo 'do thing'
            }
        }
        stage('NotOK') {
            steps {
                sh 'make fail'
            }
        }
    }
    post {
        always {
            echo 'ok'
        }
        failure {
            echo 'Failed during Which Stage?'
        }
    }
}

Answer 1

Overview

This can be achieved generically using Blue Ocean plugin API. Class PipelineNodeGraphVisitor can be used to iterate over all pipeline nodes (such as stages, parallel branches and steps). We just have to check if the type property of FlowNodeWrapper equals FlowNodeWrapper.NodeType.STAGE.

Additionally we can get the failure cause from the ErrorActions stored in the nodes.

Code

You would typically put the following code into a shared library, because it would prevent the pipeline from running in sandbox environment, if inserted directly into pipeline code.

import io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeGraphVisitor
import io.jenkins.blueocean.rest.impl.pipeline.FlowNodeWrapper
import org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper
import org.jenkinsci.plugins.workflow.actions.ErrorAction

// Get information about all stages, including the failure causes.
//
// Returns a list of maps: [[id, displayName, result, errors]]
// The 'errors' member is a list of unique exceptions.

@NonCPS
List<Map> getStageResults( RunWrapper build ) {

    // Get all pipeline nodes that represent stages
    def visitor = new PipelineNodeGraphVisitor( build.rawBuild )
    def stages = visitor.pipelineNodes.findAll{ it.type == FlowNodeWrapper.NodeType.STAGE }

    return stages.collect{ stage ->

        // Get all the errors from the stage
        def errorActions = stage.getPipelineActions( ErrorAction )
        def errors = errorActions?.collect{ it.error }.unique()

        return [ 
            id: stage.id, 
            displayName: stage.displayName, 
            result: "${stage.status.result}",
            errors: errors
        ]
    }
}

// Get information of all failed stages
@NonCPS
List<Map> getFailedStages( RunWrapper build ) {
    return getStageResults( build ).findAll{ it.result == 'FAILURE' }
}

Demo Pipeline

pipeline{
    agent any

    stages {
        stage('SuccessStage') {
            steps {
                echo 'Success'
            }
        }
        stage('FailedStage') {
            steps {
                readFile 'dfgkjsdffj'
            }
        }
        stage('SkippedStage') {
            steps {
                echo 'Skipped because of error in FailedStage'
            }
        }
    }
    post {
        failure {
            script {              
                // Print information about all failed stages
                def failedStages = getFailedStages( currentBuild )
                echo "Failed stages:\n" + failedStages.join('\n')

                // To get a list of just the stage names:
                //echo "Failed stage names: " + failedStages.displayName
            }
        }
    }
}

Blue Ocean View

Notes

If you want to get stages with other results than FAILURE, have a look at my function getFailedStages(). You can simply change the condition, e. g.:

• it.result in ['FAILURE','UNSTABLE']
  • get unstable stages aswell
• it.result != 'SUCCESS'
  • get all unsuccesfull stages, which also includes skipped stages

Possible alternative implementation:

Strictly spoken, Blue Ocean API is not necessary. It just simplifies the code alot. You can do the same using only basic Jenkins pipeline API. As a starting point, look for FlowGraphWalker for iterating over the pipeline nodes. Have a look at the code of Blue Ocean's PipelineNodeGraphVisitor to find out how they determine the "Stage" node type.

Answer 2

You can use a post directive in each stage, to act on failure with specific actions and notifications.

It's not exactly ideal as if you want that in all stages you'd have to repeat it though, and I don't think you can access your stage name dynamically, so it's really verbos and hard-coded. You could probably refactor that to use a library though.

pipeline {
    agent { label 'master'}
    stages {
        stage('Ok') {
            steps {
                echo 'do thing'
            }
            post {
                failure {
                    echo 'FAILED (in stage OK - should not happen :))'
                }
            }
        }
        stage('NotOK') {
            steps {
                sh 'make fail'
            }
            post {
                failure {
                    echo 'FAILED (in stage NotOK)'
                }
            }
        }
    }
    post {
        always {
            echo 'COMPLETED (global)'
        }
        failure {
            echo 'FAILED (global)'
        }
    }
}

Answer 3

PipelineVisitor is a fine approach. However, if you want to see just the errors, then leveraging FlowGraphTable might be even better.

The following provides a list of maps for each failed steps, and traverses the downstream jobs as well. I find it to be pretty useful.

You'll want to use a shared library to avoid the security sandbox warnings / approvals

List<Map> getStepResults() {
    def result = []
    WorkflowRun build = currentBuild()
    FlowGraphTable t = new FlowGraphTable(build.execution)
    t.build()
    for (def row in t.rows) {
        if (row.node.error) {
            def nodeInfo = [
                    'name': "${row.node.displayName}",
                    'url': "${env.JENKINS_URL}${row.node.url}",
                    'error': "${row.node.error.error}",
                    'downstream': [:]

            ]
            if (row.node.getAction(LogStorageAction)) {
                nodeInfo.url += 'log/'
            }

            for (def entry in getDownStreamJobAndBuildNumber(row.node)) {
                nodeInfo.downstream["${entry.key}-${entry.value}"] = getStepResults(entry.key, entry.value)
            }
            result << nodeInfo
        }
    }
    log(result)
    return result
}

Map getDownStreamJobAndBuildNumber(def node) {
    Map downStreamJobsAndBuilds = [:]
    for (def action in node.getActions(NodeDownstreamBuildAction)) {
        def result = (action.link =~ /.*\/(?!\/)(.*)\/runs\/(.*)\//).findAll()
        if (result) {
            downStreamJobsAndBuilds[result[0][1]] = result[0][2]
        }
    }
    return downStreamJobsAndBuilds
}

Answer 4

Instead of adding post section in every stage, I found some solution that shouldn't be working in Declarative Pipeline from my point of view, but it does. All is you need is to override stage:

def stage(String name, Closure cl) {
    echo "Stage: ${name}"
    try {
        cl()
    } catch (Exception e) {
        // I needed to save failed stage and message for parent pipeline job
        // so I saved them in environment variables, otherwise it can be saved
        // in global variables
        if (!env.FAILED_STAGE) {
            env.FAILED_STAGE = name
            env.FAILED_MESSAGE = e.getMessage()
        }
    }
}

pipeline {

    options { timestamps() }
    agent { label 'master' }
    stages {
        stage('First stage') {
            steps {
                //Any steps are working
                script {
                    sh "echo first"
                }
            }
        }
        stage('Second stage') {
            steps {
                echo "second"
            }
        }
        stage('Fail stage') {
            steps {
                error "failed"
            }
        }
        stage('Final stage') {
            steps {
                build "Other job"
            }
        }
    }
    post {
        failure {
            echo "Failed stage: ${env.FAILED_STAGE}"
            echo "Error message: ${env.FAILED_MESSAGE}"
        }
    }
}

The most strange thing to me is that after stage failure other stages are skipped as they should. Here is the output:

14:05:14 Stage: First stage
[Pipeline] script
[Pipeline] {
[Pipeline] sh
14:05:14 + echo first
14:05:14 first
[Pipeline] }
[Pipeline] // script
[Pipeline] echo
14:05:14 Stage: Second stage
[Pipeline] echo
14:05:14 second
[Pipeline] echo
14:05:14 Stage: Fail stage
[Pipeline] error
[Pipeline] error
[Pipeline] echo
14:05:14 Stage: Final stage
Stage "Final stage" skipped due to earlier failure(s)
[Pipeline] echo
14:05:14 Stage: Declarative: Post Actions
[Pipeline] echo
14:05:14 Failed stage: Fail stage
[Pipeline] echo
14:05:14 Error message: failed
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: failed
Finished: FAILURE

EDIT: Note, that you will lose stage view, as there will be no normal stages from Jenkins point of view.

Answer 5

I built something a bit simpler for my own needs, although it still requires the snippet in each stage.

I came here looking for a way to not repeat the error-getting in the post section of every stage, and I love the above solutions. I may incorporate bits from the other answers into my library at some point.

I have an env.informationalMessages var that is sent along with the notification emails at the end of every run.

In each stage you do an Unsuccessful post section. This should capture failed, aborted, unstable results. Cleanup runs after any other post conditions.

stage ('some stage name') {
  steps { ... }  
  post {
    unsuccessful {
      addStageResultToInfoMessages()
    }
    cleanup  {
    // whatever else you want to do
    }
  }
}

vars/addStageResultToInfoMessages.groovy:

// Adds a message with the stage result into the informationalMessages env var.
def call() {
  addToInfoMessages("${currentBuild.result} in stage '${STAGE_NAME}'.")
}

vars/addToInfoMessages.groovy

// Adds the passed-in string to the informationalMessages env var
// that gets sent with notification emails.
// env.informationalMessages is initialized to '' at the start of each run.
def call(String message) {
  env.informationalMessages += "${message}\n"
}

Then at the end of the pipe, another post section:

post {
  unsuccessful {
    addToInfoMessages(getErrorMessage())
  }
  cleanup {
    notifyEmail()
  }
}

vars/getErrorMessage.groovy grabs the raw console text from the Jenkins API and looks for the text 'ERROR:'. The outdated Groovy in Jenkins doesn't support null-safe navigation on lists, so you have to do it old-school.

// gets the Error message from the run's console text
// uses the jenkins api
def call() {
  // get the raw text of the build's console output
  response = httpRequest ignoreSslErrors: true, url: "${BUILD_URL}consoleText"
  
  // find lines with 'ERROR:'
  err = response.getContent().findAll(/.*ERROR:.*/)

  // if null-safe, split off the time code
  if (err) { return err[-1].split('Z] ')[1] }
  else { return 'Error not found' }
}

Just another way to do this.