Run Jenkins Stage on Master Executor

Question

I am trying to create a pipeline that builds multiple projects for an application.

When a project succeeds building (in parallel), I need to update a file on the master executor (to change a tag). However, Jenkins gets stuck 'waiting for executors' (I guess because the executor is waiting for the parallel stages to finish?)

My pipeline (simplified):

def parallelStages = [:]
pipeline {
    agent any
    options { skipDefaultCheckout() }
    stages {
        // Grab code from GitRepo
        stage ("Checkout Code") {
            steps {
               // grab code from repo
               // stash code for slaves
               // This stage is executed on master, at workspace NAME@2
            }
        }
        stage ("Find Project") {
            steps {
               // findFiles is used to find projects, and assign them 
               parralelStages.put('foldername', BuildAndTestProject('folder'))
               // This stage is executed on master, at workspace NAME@2
            }
        }
        stage ("BuildAndTest") {
            steps {
                script {
                    parralelStages.put('failFast', false)
                    parallel(parralelStages)
                }
            }
        }
        stage ("deploy") {
            // Deploy newly built projects based on YAML
        }
    }

The BuildAndTestProject-Function (simplified):

def BuildAndTestReact(def name) {
    def label = "build-slave-${name}-${BUILD_NUMBER}"
    return {
        podTemplate(label: label, yaml: """) {
            node(label) { // This is a build-slave
                stage("BuildAndTest") {
                    stage("Unstash") {
                        // Unstash source for project on slave
                    }
                    stage("Build") {
                        // Build Project
                    }
                    stage("Test") {
                        // Test Project
                    }
                    stage("Set Yaml-Tag") {
                        node('master') { // THIS SHOULD RUN ON THE MAIN EXECUTOR, AS IT HAS THE FILES FOR ALL PROJECTS (I don't really know how I would dynamically stash/unstash these back to that executor from this slave)
                            // Update YAML for this project to set the tag to the one that was built in Build-Stage
                         }
                     }
                 }
             }
         }
     }
 }

What I want is for the main executor (the one running the 'Checkout Code'-stage) to update the YAML for every project that was built in parallel.

I don't care if the YAML-update happens sequentially, I just need it to happen for all of the projects that were built.

In the future, I want to make sure that only projects with file-changes are grabbed in the 'Find Project'-stage. All other projects will then keep the tag 'latest' for the 'deploy'-stage.

With multiple executors (on the main-node) the 'Set Yaml-Tag' is run in different workspaces (e.g. NAME@3 NAME@4 etc) which don't actually have the files that are in NAME@2. (Meaning it isn't applied to the files)

EDIT: Visual Aid (Courtesy of Blue Ocean plugin):

I need the Update YAML-stages (circled in red) to run on main executor (circled in blue) because I apply the files in 'Clear Test Namespace'

MaratC MaratC · Accepted Answer · 2020-06-07T08:48:01

This behaviour is exactly by design, since Jenkins does not want to update the same file when several branches run in parallel, because in the vast majority of the scenarios one branch would break the other.

You may want to address your issue in a different way. One I can think of would be to make all the parallel branches set an environment variable with the value you want, and make a final branch read these environment variables and update the file with the values, once all parallel branches have finished.

Run Jenkins Stage on Master Executor

1 Answers