How to run multiple stages on the same node with declarative Jenkins pipeline?

45
votes

Goal
Run multiple stages of a declarative Jenkins pipeline on the same node.

Setup
This is just a minimal example to show the problem. There are 2 Windows nodes "windows-slave1" and "windows-slave2" both labeled with the label "windows".

NOTE: My real Jenkinsfile cannot use a global agent because there are groups of stages that require to run on different nodes (e.g. Windows vs. Linux).

Expected Behaviour
Jenkins selects one of the nodes in "Stage 1" based on the label and uses the same node in "Stage 2" because the variable windowsNode was updated to the node selected in "Stage 1".

Actual Behaviour
"Stage 2" sometimes runs on the same and sometimes on a different node than "Stage 1". See the output below.

Jenkinsfile

#!groovy

windowsNode = 'windows'

pipeline {
  agent none
  stages {
    stage('Stage 1') {
      agent {
        label windowsNode
      }
      steps {
        script {
          // all subsequent steps should be run on the same windows node
          windowsNode = NODE_NAME
        }
        echo "windowsNode: $windowsNode, NODE_NAME: $NODE_NAME"
      }
    }
    stage('Stage 2') {
      agent {
        label windowsNode
      }
      steps {
        echo "windowsNode: $windowsNode, NODE_NAME: $NODE_NAME"
      }
    }
  }
}

Output

[Pipeline] stage
[Pipeline] { (Stage 1)
[Pipeline] node
Running on windows-slave2 in C:\Jenkins\workspace\test-agent-allocation@2
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] echo
windowsNode: windows-slave2, NODE_NAME: windows-slave2
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Stage 2)
[Pipeline] node
Running on windows-slave1 in C:\Jenkins\workspace\test-agent-allocation
[Pipeline] {
[Pipeline] echo
windowsNode: windows-slave2, NODE_NAME: windows-slave1
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
Finished: SUCCESS

Any ideas what's wrong with the setup? I guess it's how the Jenkinsfile is parsed and executed.

Other suggestions? Maybe there is a Jenkins API to select a node based on the "windows" label when setting windowsNode initially.

4
jenkinsjenkins-pipeline
Your stage 2 is not picking up the renamed variable. If you change windowsNode = 'windows' to windowsNode = 'asdf' then you should see a "There are no nodes with the label ‘asdf’" error, meaning your stage 2 is still running label 'windows' and not label 'windows-slave2'. – Quantic
However, I tried stage("Stage 2" ) { environment { someVariable = "$windowsNode" } agent { label env.someVariable } .. and although it allocates a node and gives no "missing property" errors (meaning agent at least is aware that someVariable exists), it does NOT work. If I print the value of env.someVariable it is the node name, so although it can see the correct windowsNode variable, I suspect that someVariable is in some blank state in the agent directive causing it to run something like agent label '' instead of agent label 'windows-slave2'. – Quantic
I also tried environment { someVariable = "\'$windowsNode\'" } which makes the variable 'windows-slave2' instead of just windows-slave2, but it still randomly allocates between my two test nodes (still no errors about the variable or label name though). – Quantic

4 Answers

30
votes

Since version 1.3 of Declarative Pipeline plugin, this is officially supported. It's officially called "Sequential Stages".

pipeline {
    agent none

    stages {
        stage("check code style") {
            agent {
                docker "code-style-check-image"
            }
            steps {
                sh "./check-code-style.sh"
            }
        }

        stage("build and test the project") {
            agent {
                docker "build-tools-image"
            }
            stages {
               stage("build") {
                   steps {
                       sh "./build.sh"
                   }
               }
               stage("test") {
                   steps {
                       sh "./test.sh"
                   }
               }
            }
        }
    }
}

Official announcement here: https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/

11
votes

You could define stages inside a script block. Those stages are kind of sub-stages of a parent stage running in a given agent. That was the approach that I had to use in a similar use case than yours.

#!groovy

windowsNode = 'windows'

pipeline {
  agent none
  stages {
    stage('Stage A') {
      agent {
        label windowsNode
      }
      steps {
        script {

          stage('Stage 1') {        
            windowsNode = NODE_NAME
            echo "windowsNode: $windowsNode, NODE_NAME: $NODE_NAME"
          }

          stage('Stage 2') {
            echo "windowsNode: $windowsNode, NODE_NAME: $NODE_NAME"
          }
        }
      }
    }
  }
}
5
votes

I have found that this works as you would expect

#!groovy

windowsNode = 'windows'

pipeline {
    agent none
    stages {
        stage('Stage 1') {
            steps {
                node(windowsNode) {
                    script {
                        // all subsequent steps should be run on the same windows node
                        windowsNode = NODE_NAME
                    }
                    echo "windowsNode: $windowsNode, NODE_NAME: $NODE_NAME"
                }
            }
        }
        stage('Stage 2') {
            steps {
                node(windowsNode) {
                    echo "windowsNode: $windowsNode, NODE_NAME: $NODE_NAME"
                }
            }
        }
    }
}
-6
votes

replace agent none with agent any