Jenkins Build Pipeline - Restart At Stage

Question

I have the following build pipeline set up as a job:

Stage 1 - verify all dependencies exist
Stage 2 - build the new jar
Stage 3 - Run integration tests
Stage 4 - Deploy to staging environment (manual step)
Stage 5 - Deploy to production environment (manual step)

I am looking for a way to start the build pipeline from a particular stage in case of a transient failure. For example, let's say there was a network issue when the user clicked to deploy to production. I don't think it makes sense to start the pipeline from stage 1... I'd like to try that step again and continue on from there in the pipeline. I don't see any functionality like this in the Build Pipeline Plugin.

Thanks!!

Answer 1

checkpoint is what you are looking for. Unfortunately it is only available in the CloudBees Jenkins Enterprise suite, not in the free version.

Let's hope it makes it into the open-source version as it seems to be a very common use case.

Answer 2

Meanwhile, many other answers are obsolete, as Jenkins provides a built-in solution that allows you to restart a job from any stage: https://jenkins.io/doc/book/pipeline/running-pipelines/ It is applicable to declarative pipeline only through.

Answer 3

A better solution is a solution similar to what I suggested in this question:

Write a pipelining script that has has "if"-guards around the single stages, like this:

stage "s1"
if (theStage in ["s1"]) {
    sleep 2
}

stage "s2"
if (theStage in ["s1", "s2"]) {
    sleep 2
}

stage "s3"
if (theStage in ["s1", "s2", "s3"]) {
    sleep 2
}

Then you can make a "main" job that uses this script and runs all stages at once by setting the parameter "theStage" to "s1". This job will collect the statistics when all stages are run at once and give you useful estimation times.

Furthermore, you can make a "partial run" job that uses this script and that is parametrized with the stage that you want to start with. The estimation will not be very useful, though.

Answer 4

You can wrap your code in a retry step:

stage "Deployment"
retry(3) {
  sh "deploy.."
}

EDIT: This might help in the free version of Jenkins. Users of CloudBees Enterprise, please see @tarantoga's answer.

Answer 5

Here's another sketch to run stages conditionally without breaking the Stage View Plugin history.

As they say:

Dynamic stages: in general, if you want to visualize dynamically changing stages, make it conditional to execute the stage contents, not conditional to include the stage

Here's what I've come up with so far. Seems to work mostly: (Just ignore the other dummy steps)

We define a little conditionalStage helper function that neatly wraps up the stage name checking from the JP_STAGE Jenkins Job parameter.

Notice how conditionalStage first opens the stage and then checks stageIsActive within the stage, just skipping all steps. This way, the Stage View Plugin sees all stages and soesn't mess up, but the stages' steps are still skipped.

def stageSelect = JP_STAGE.toLowerCase()

// test if stage or any of sub-stages is active
def stageIsActive(theStage, theStages) { 
    // echo "pass: $theStages"
    // ARGL: https://issues.jenkins-ci.org/browse/JENKINS-26481
    // def lcStages = theStages.collect {it.toLowerCase()}
    def lcStages = []
    for (def s : theStages) { lcStages += s.toLowerCase() }
    def lcAllStages = lcStages + ['all']
    // echo "check: $lcAllStages"
    // echo JP_STAGE.toLowerCase()
    if (JP_STAGE.toLowerCase() in lcAllStages) {
        echo "Run: Stage '$theStage' is active through '$JP_STAGE'."
        return true
    } else {
        echo "Skip: Stage '$theStage' is NOT active through '$JP_STAGE'."
        return false
    }
}

// 1st element should be the stage, optionally followed by all sub-stages
def conditionalStage(names, stageBody) {
  stage(names[0]) { if (stageIsActive(names[0], names)) {
    stageBody()
  }}  
}

timestamps {
// --S--

conditionalStage(['Intro']) { 
    echo 'Outside Node'

    build job: 'FreeX', wait: true
    sleep 3
}

// --S--
conditionalStage(['AtNode', 'Hello', 'Done']) {
    node {
        // Cloudbees Enterprise Only: checkpoint 'Now'
        conditionalStage(['Hello']) {
            echo 'Hello World @ Node'
            sleep 4
        }
        conditionalStage(['Done']) {
            dir('C:/local') {
                echo pwd()
            }
        }
    }
}

}//timestamps

Answer 6

A bit old topic but since Jenkins still (!) doesn't support this I'm sending another solution for scripted pipeline implementations. It's based on building stages list dynamically when running pipeline.

1. step - stages definition enum

enum Steps {
  PREPARE(0, "prepare"), 
    BUILD(1, "build"), 
    ANALYSE(2, "analyse"), 
    CHECKQG(3, "checkQG"), 
    PROVISION(4, "provision"), 
    DEPLOY(5, "deploy"), 
    ACTIVATE(6, "activate"), 
    VERIFY(7, "verify"), 
    CLEANUP(8, "cleanup")

  Steps(int id, String name) {
      this.id = id
          this.name = name
  }

  private final int id
    private final String name

  int getId() {
      id
  }

      String getName() {
      name
  }

    public static Steps getByName(String name) {
        println "getting by name " + name
        for(Steps step : Steps.values()) {
          if(step.name.equalsIgnoreCase(name)) { 
              return step 
          }
        }
        throw new IllegalArgumentException()
    }
}

2. method creating the final steps list

  def prepareStages(def startPoint){
        println "preparing build steps starting from " + startPoint
        Set steps = new LinkedHashSet()
        steps.add(Steps.PREPARE)
        steps.add(Steps.BUILD)
        steps.add(Steps.ANALYSE)
        steps.add(Steps.CHECKQG)
        steps.add(Steps.PROVISION)
        steps.add(Steps.DEPLOY)
        steps.add(Steps.ACTIVATE)
        steps.add(Steps.VERIFY)
        steps.add(Steps.CLEANUP)
        List finalSteps = new ArrayList()
        steps.each{
            step ->
                if (step.id >= startPoint.id) {
                    finalSteps.add(step)
                }
        }
        return finalSteps
    }

3. and u can use it like this

def stages = prepareStages(Steps.getByName("${startStage}"))

node {
    try {
        //pipelineTriggers([pollSCM('${settings.scmPoolInterval}')])  //this can be used in future to get rid build hooks 

        sh "echo building " + buildVersionNumber(${settings.isTagDriven})
        tool name: 'mvn_339_jenkins', type: 'maven'

        script {             
            println "running: " + stages
        }

        stage('Prepare') {
            if (stages.contains(Steps.PREPARE)) {
                script { currentStage = 'Prepare' }
               //.....
            }
        } //...

the "startStage" is a build parameter defined as follows

parameters { choiceParam('startStage', [ 'prepare', 'build', 'analyse', 'checkQG', 'provision', 'deploy', 'activate', 'verify', 'cleanup' ], 'Pick up the stage you want to start from') }

This allows me to pick up the stage I want to start the pipeline from (prepare stage is set by default)

Answer 7

What you could do is to put the single steps into groovy scripts. Then you can make a "runAll"-job that loads all of the scripts in the correct order, and single jobs for the different steps.

Although this is a way that should work, I do not think that this is the ideal solution, as it means that you have to take care how the different steps exchange information, so that the steps can run independently.

A built-in solution would be much better.