Common setup for parallel Jenkins pipeline stages

Question

I'm trying to setup a Jenkins build that syncs SVN to a given changelist and then builds code for various different configurations in parallel. I have a prebuild step that needs to be run once per agent regardless of how many builds that agent will build. My current declarative pipeline looks like this:

pipeline {
agent none
stages
{   
    stage('parallel')
    {
        failFast true

        parallel
        {
            stage('Release')
            {
                agent {
                    label {
                        label "windows"
                    }
                }                   
                steps
                {
                    checkout([$class: 'SubversionSCM',
                    additionalCredentials: [],
                    excludedCommitMessages: '',
                    excludedRegions: '',
                    excludedRevprop: '',
                    excludedUsers: '',
                    filterChangelog: false,
                    ignoreDirPropChanges: false,
                    includedRegions: '',
                    locations: [[credentialsId: 'SVN',
                        depthOption: 'infinity',
                        ignoreExternalsOption: false,
                        local: '.',
                        remote: 'https://my-repo']],
                    workspaceUpdater: [$class: 'UpdateUpdater']])

                    bat 'Prebuild.bat'
                    bat 'msbuild Game/Code/Application.sln "Release|x64"'
                }
            }
            stage('Debug Optimized')
            {
                agent {
                    label {
                        label "windows"
                    }
                }
                steps
                {
                    checkout([$class: 'SubversionSCM',
                    additionalCredentials: [],
                    excludedCommitMessages: '',
                    excludedRegions: '',
                    excludedRevprop: '',
                    excludedUsers: '',
                    filterChangelog: false,
                    ignoreDirPropChanges: false,
                    includedRegions: '',
                    locations: [[credentialsId: 'SVN',
                        depthOption: 'infinity',
                        ignoreExternalsOption: false,
                        local: '.',
                        remote: 'https://my-repo']],
                    workspaceUpdater: [$class: 'UpdateUpdater']])

                    bat 'Prebuild.bat'
                    bat 'msbuild Game/Code/Application.sln "Debug Optimized|x64"'
                }
            }
        }
    }
}
}

While this works a) it's unmaintainable (I have about 20 configs to build in reality, and I don't want to maintain 20 copies of my scm settings), b) it's running the checkout/prebuild once per stage rather than once per agent. I notice there is a default checkout for agents, but as far as I can tell this doesn't work with SVN.

Is there a better way of doing parallel stages than this?

You could do the checkout in one stage and stash the sources. Then unstash them as the first step in the parallel stages. I've done that using a scripted pipeline though. — pitseeker
My branch is too big (10s of GB) for that to be viable, unfortunately — Iain Brown
Not sure about the declarative pipeline, but at least with the scripted pipeline you can checkout and then do further stuff in a single stage (or in subsequent stages). You can call "parallel" over a set of "nodes" that each execute a sequence of stages. Concerning the maintenance of the scm-settings: perhaps you can move those to a shared library. See jenkins.io/doc/book/pipeline/shared-libraries — pitseeker

Luc Luc · Accepted Answer · 2018-04-24T13:56:25

did you heard about Jenkins Multijob plugin? https://wiki.jenkins.io/display/JENKINS/Multijob+Plugin

Could it fit your need?

You have also the possibility to build downstream job, when one of your job is finished.

Common setup for parallel Jenkins pipeline stages

1 Answers