Running multiple stages in parallel in the same node with the same workspace

Question

I have 3 build jobs which run in parallel in a declarative jenkinsfile. They run in the same node and need to use the same workspace. The issue is the workspace which Jenkins refers for each stage, for example:

C:\UserData\Workspace                 \\Workspace for Job1
C:\UserData\Workspace@2               \\Workspace for Job2
C:\UserData\Workspace@3               \\Workspace for Job3

Jenkins appends '@2' and '@3' for the remaining 2 stages and hence there is a path issue and job fails. Can someone help me in resolving this issue?

My code is:

pipeline {
    stages {
    stage('Build') {
                parallel {
                    stage('Job1') {
                        agent {
                              node {
                                    label 'label1'
                                    customWorkspace = "C:\UserData\Workspace"
                }
            }     
                   stage('Job2') { ... similar code ... }
                   stage('Job3') { ... similar code ... }
    }
}

Potaito Potaito · Accepted Answer · 2019-12-03T08:31:45

The different workspaces are necessary, because otherwise the jobs would run inside the same workspace at the same time and encounter resource conflicts. What if two or more jobs read from or write to the same file? This is dangerous and the results are unpredictable. Also note that there is no guarantee which parallel job will finish first. It might be that your pipeline works fine on one worker node, but would fail on another with a different configuration (max number of concurrent jobs for example). I see two scenarios here:

Your parallel jobs depend on each other, in which case they should be sequential according to the depenceny. If some files are needed from a previous job, you can use the stash/unstash commands to obtain these files.
Your parallel jobs do not depend on each other, in which case it should be totally fine if they run in separate workspaces.

If you explain why the jobs "need to use the same workspace", we can perhaps find a better solution.

Running multiple stages in parallel in the same node with the same workspace

1 Answers