How to re-use previously created workspace across stages

Question

I am facing an issue where I have two stages defined in my pipeline that both are run on the same node and need to be run in the same workspace.

The first of these stages runs on my master node initially, but towards the end of the steps defined has to unstash some files onto a different node.

The second stage then just needs to continue on my master, and relies on some modules that were installed from the first stage.

Here is my pipeline to better explain:

#!groovy
pipeline {
  agent { label 'master' }
  stages {
    stage('Build') { // 1. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
      steps {
        sh '''
          npm install
          bower install
          gulp set-staging-node-env
          gulp prepare-staging-files
          gulp webpack
        '''
        stash includes: 'dist/**/*', name: 'builtSources'
        stash includes: 'config/**/*', name: 'appConfig'
        node('Protractor') { // 2. Running on vnccentos7 in /var/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
          dir('/opt/foo/deploy/') {
            unstash 'builtSources'
            unstash 'appConfig'
          }
        }        
      }
    }
    stage('Unit Tests') {
      agent { label 'master' } // 3. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2
      steps {
        parallel (
          "Jasmine": {
            sh 'gulp karma-tests-ci'
          },
           "Mocha": {
            sh 'gulp mocha-tests'
          }
        )
      }
    }
  }
}

As you can see, I have added comments at the start of each stage\node used to show the jenkins output I see for what workspaces get allocated.

The problem I am facing is that the Unit Tests stage fails as it tries to use some node modules it cannot find. These are present in the first workspace that gets created which is where I want this stage to continue using, so not using a new '@2' suffixed workspace.

Is there a way to tell the Jenkins to preserve previously created workspaces in the pipeline?

EDIT

I am guessing as I have specified agent {label:'master'} again in my next stage, this is what cases a new workspace to get created? Should I be using the node approach instead? Would that allow the same workspace to get used?

I actually have tried using node('master'){...} around each of the parallel steps in my Unit Tests stage, but these still use the @2 suffixed workspace rather than the original.

I have seen other threads talking about how you should not re-use the same workspace as you could run into issues with file locks. They suggest instead to archive\unarchive the workspace between steps.

I have also seen approaches where you can store the workspace path in a variable and use this later on which sounds good for my case, but I havent found any declarative syntax samples, only groovy ones.

EDIT 2

I have now tried a few approaches involving saving the allocated workspace from the first stage into a variable and using in a ws(...) directive in the later stages:

pipeline {
  agent { label 'master' }
  stages {
    stage('Build') {
      steps {
        script {
          def workspace = pwd()
        }
        sh '''
          npm install
          bower install
          gulp set-staging-node-env
          gulp prepare-staging-files
          gulp webpack
        '''
        stash includes: 'dist/**/*', name: 'builtSources'
        stash includes: 'config/**/*', name: 'appConfig'
        node('Protractor') {
          dir('/opt/foo/deploy/') {
            unstash 'builtSources'
            unstash 'appConfig'
          }
        }        
      }
    }
    stage('Unit Tests') {
      steps {
        parallel (
          "Jasmine": {
            node('master') {
              ws("${workspace}"){
                sh 'gulp karma-tests-ci'
              }
            }
          },
          "Mocha": {
            node('master') {
              ws("${workspace}"){
                sh 'gulp mocha-tests'
              }
            }
          }
        )
      }
      post {
        success {
          sh 'gulp combine-coverage-reports'
          sh 'gulp clean-lcov'
          publishHTML(target: [
            allowMissing: false,
            alwaysLinkToLastBuild: false,
            keepAll: false,
            reportDir: 'test/coverage',
            reportFiles: 'index.html',
            reportName: 'Test Coverage Report'
          ])
        }
      }
    }
    }
}

I did try just removing the second agent declaration from the Unit Tests stage but the stage remained on my Protractor node which I didnt want it to do. So following the answers\comments here I then used node blocks around each of my parallel steps and used the ws blocks as you can see,

The stage fails, and I can see from the logs that its not using the workspace allocated from the first stage (without the @ suffixes on):

[Jasmine] Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2
[Pipeline] [Jasmine] {
[Pipeline] [Jasmine] ws
[Jasmine] Running in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2@2
[Pipeline] [Jasmine] {
[Pipeline] [Jasmine] sh
[Jasmine] [_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2@2] Running shell script
[Jasmine] + gulp karma-tests-ci
[Jasmine] [08:27:01] No gulpfile found

Its even double suffixing with @2, so I'm unsure what its doing now.

Answer 1

Not sure if it fits your use case, but this example script shows how to share the same node/workspace between different stages & containers:

Additionally, if you're running a Docker agent for a specific stage while specifying agent { label 'whatever' } at the top level, you can ensure that this stage will use the same node and workspace as the rest of the Pipeline:

pipeline {
  agent {
    label 'whatever'
  }
  stages {
    stage('build') {
      steps {
        sh "./build-artifact.sh"
      }
    }
    stage('test in docker') {
      agent {
        docker {
          image 'ubuntu:16.04'
          reuseNode true
        }
      }
      steps {
        sh "./run-tests-in-docker.sh"
      }
    }
  }
}

https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Controlling-your-build-environment#reusing-nodeworkspace-with-per-stage-docker-agents

Answer 2

Use dir instead of ws.

ws automatically amend the prefix "@NUMBER" when the workspace to be used was already occupied by other build job.

dir just move current working directory to exactly where you designate.

pipeline {
    agent none
    environment {
        WIN_WORKSPACE = ""
        MAC_WORKSPACE = ""
    }
    stages {
        stage("Build") {
            parallel {
                stage("Build on Windows") {
                    agent {
                        label "windows"
                    }
                    steps {
                        script {
                            WIN_WORKSPACE = WORKSPACE
                        }
                        // steps...
                    }
                }
                stage("Build on macOS") {
                    agent {
                        label "macos"
                    }
                    steps {
                        script {
                            MAC_WORKSPACE = WORKSPACE
                        }
                        // steps...
                    }
                }
            }
        }
        stage("Deploy") {
            parallel {
                stage("Deploy on Windows") {
                    agent {
                        label "windows"
                    }
                    steps {
                        dir(WIN_WORKSPACE) {
                            // steps...
                        }
                    }
                }
                stage("Deploy on macOS") {
                    agent {
                        label "macos"
                    }
                    steps {
                        dir(MAC_WORKSPACE) {
                            // steps...
                        }
                    }
                }
            }
        }
    }
}

Works well as you want.

Answer 3

You can specify a global agent but you must keep in mind that for each underlying agent directive a new workspace will be used.

The docker agents have a usefull option telling reuseNode to keep using the same node/workspace.

Example 1: Using "agent none", 2 stages use the some workspace

pipeline {
    agent none // <- the difference is here
    stages {
        stage('prep') {
            agent { docker { image 'yourdockerimage' }}
            steps {
                sh 'pwd' // same workspace, without @2
            }
        }
        stage('build') {
            agent { label 'master' }
            steps {
                sh 'pwd' // same workspace, without @2
            }
        }
    }
}

Example 2: Using a global agent

pipeline {
    agent { label 'master' } // <- specify the global agent
    stages {
        stage('prep') {
            agent { docker { image 'yourdockerimage' }} // <- do not ask to reuse the node/workspace
            steps {
                sh 'pwd' // different workspace, with @2 appended
            }
        }
        stage('build') {
            // no agent specified, use global agent
            steps {
                sh 'pwd' // same workspace, without @2
            }
        }
        stage('publish') {
            agent { docker { image 'yourdockerimage' reuseNode true }}
            steps {
                sh 'echo "published"' // same workspace, without @2
            }
        }
    }
}

Answer 4

Specify a custom workspace. From the Pipeline syntax: "ws: Allocate workspace"

ws("/usr/local/jenkins/jobs/custom_workspace") {
    stage . . . 
    stage . . . 
}

. . does this work ?

Answer 5

I am glad to say that, Jenkins can share workspace now.

You can specify stages nested within other stages to share workspace.

The new feature is called Sequential Stages.

I will use your original code to implement it.

#!groovy
pipeline {
  agent { label 'master' }
  stages {
    stage('Build') { // 1. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
      steps {
        sh '''
          npm install
          bower install
          gulp set-staging-node-env
          gulp prepare-staging-files
          gulp webpack
        '''
        stash includes: 'dist/**/*', name: 'builtSources'
        stash includes: 'config/**/*', name: 'appConfig'
        node('Protractor') { // 2. Running on vnccentos7 in /var/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ
          dir('/opt/foo/deploy/') {
            unstash 'builtSources'
            unstash 'appConfig'
          }
        }        
      }
    }
    stage('Unit Tests') { // 3. Running on master in /var/lib/jenkins/workspace/_Pipelines_IACT-Jenkinsfile-UL3RGRZZQD3LOPY2FUEKN5XCY4ZZ6AGJVM24PLTO3OPL54KTJCEQ@2
        parallel {
            stage("Unit Tests@") {
                agent {label 'master'}
                stages {
                    stage("Jasmine") {
                        steps {
                            sh 'gulp karma-tests-ci'
                        }
                    }
                    stage(Mocha") {
                        steps {
                            sh 'gulp mocha-tests'
                        }
                    }
                }
            }
        }
    }
  }
}

Answer 6

I thing using the node syntax will solve it.

in order to be certain I would use a dir scope to set the workspace on your own

Answer 7

The External Workspace Manager Plugin might solve your problem.

[...] it defines the local path to the workspace, and switches to it.

// Basic usage:

def extWorkspace = exwsAllocate diskPoolId: 'diskpool1'
node ('linux') {
    exws (extWorkspace) {
        scm checkout
        sh 'mvn clean install -DskipTests'
    }
}
node ('test') {
    exws (extWorkspace) {
        sh 'mvn test'
    }
}