Use a lightweight executor for a declarative pipeline stage (agent none)

Question

I'm using Jenkins Pipeline with the declarative syntax, currently with the following stages:

1. Prepare
2. Build (two parallel sets of steps)
3. Test (also two parallel sets of steps)
4. Ask if/where to deploy
5. Deploy

For steps 1, 2, 3, and 5 I need and agent (an executor) because they do actual work on the workspace. For step 4, I don't need one, and I would like to not block my available executors while waiting for user input. This seem to be referred to as either a "flyweight" or "lightweight" executor for the classic, scripted syntax, but I cannot find any information on how to achieve this with the declarative syntax.

So far I've tried:

1. Setting an agent directly in the pipeline options, and then setting agent none on the stage. This has no effect, and the pipeline runs as normalt, blocking the executor while waiting for input. It is also mentioned in the documentation that it will have no effect, but I thought I'd give it a shot anyway.
2. Setting agent none in the pipeline options, and then setting an agent for each stage except #4. Unfortunately, but expectedly, this allocates a new workspace for every stage, which in turn requires me to stash and unstash. This is both messy and gives me further problems in the parallel stages (2 and 3) because I cannot have code outside the parallel construct. I assume the parallel steps run in the same workspace, so stashing/unstashing in both would have unfortunate results.

Here is an outline of my Jenkinsfile:

pipeline {
    agent {
        label 'build-slave'
    }
    stages {
        stage("Prepare build") {
            steps {
                // ...
            }
        }
        stage("Build") {
            steps {
                parallel(
                    frontend: {
                        // ...
                    },
                    backend: {
                        // ...
                    }
                )
            }
        }
        stage("Test") {
            steps {
                parallel(
                    jslint: {
                        // ...
                    },
                    phpcs: {
                        // ...
                    },
                )
            }
            post {
                // ...
            }
        }
        stage("Select deploy target") {
            steps {
                script {
                    // ... code that determines choiceParameterDefinition based on branch name ...
                    try {
                        timeout(time: 5, unit: 'MINUTES') {
                            deployEnvironment = input message: 'Deploy target', parameters: [choiceParameterDefinition]
                        }
                    } catch(ex) {
                        deployEnvironment = null
                    }
                }
            }
        }
        stage("Deploy") {
            when {
                expression {
                    return binding.variables.get("deployEnvironment")
                }
            }
            steps {
                // ...
            }
        }
    }
    post {
        // ...
    }
}

Am I missing something here, or is it just not possible in the current version?

Answer 1

Setting agent none at the top level, then agent { label 'foo' } on every stage, with agent none again on the input stage seems to work as expected for me.

i.e. Every stage that does some work runs on the same agent, while the input stage does not consume an executor on any agent.

pipeline {
    agent none
    stages {
        stage("Prepare build") {
            agent { label 'some-agent' }
            steps {
                echo "prepare: ${pwd()}"
            }
        }
        stage("Build") {
            agent { label 'some-agent' }
            steps {
                parallel(
                    frontend: {
                        echo "frontend: ${pwd()}"
                    },
                    backend: {
                        echo "backend: ${pwd()}"
                    }
                )
            }
        }
        stage("Test") {
            agent { label 'some-agent' }
            steps {
                parallel(
                    jslint: {
                        echo "jslint: ${pwd()}"
                    },
                    phpcs: {
                        echo "phpcs: ${pwd()}"
                    },
                )
            }
        }
        stage("Select deploy target") {
            agent none
            steps {
                input message: 'Deploy?'
            }
        }
        stage("Deploy") {
            agent { label 'some-agent' }
            steps {
                echo "deploy: ${pwd()}"
            }
        }
    }
}

However, there are no guarantee that using the same agent label within a Pipeline will always end up using the same workspace, e.g. as another build of the same job while the first build is waiting on the input.

You would have to use stash after the build steps. As you note, this cannot be done normally with parallel at the moment, so you'd have to additionally use a script block, in order to write a snippet of Scripted Pipeline for the stashing/unstashing after/before the parallel steps.

Answer 2

There is a workaround to use the same build slave in the other stages. You can set a variable with the node name and use it in the others.

ie:

pipeline {
    agent none
    stages {
        stage('First Stage Gets Agent Dynamically') {
            agent {
                node {
                    label "some-agent"
                }
            }
            steps {
                echo "first stage running on ${NODE_NAME}"
                script {
                  BUILD_AGENT = NODE_NAME
                }
            }
        }
        stage('Second Stage Setting Node by Name') {
            agent {
                node {
                    label "${BUILD_AGENT}"
                }
            }
            steps {
                echo "Second stage using ${NODE_NAME}"
            }
        }
    }
}

Answer 3

As of today (2021), you can use nested stages (https://www.jenkins.io/doc/book/pipeline/syntax/#sequential-stages) to group all the stages that must run in the same workspace before the input step, and all the stages that must be run in the same workspace after the input step. Of course, you need to stash or to store artifacts in some external repository before the input step, because the second workspace may not be the same than the first one:

pipeline {
    agent none
    stages {
        stage('Deployment to Preproduction') { 
            agent any 
            stages {
                stage('Stage PRE.1') { 
                    steps {
                        echo "StagePRE.1"
                        sleep(10)
                    }
                }
                stage('Stage PRE.2') { 
                    steps {
                        echo "Stage PRE.2"
                        sleep(10)
                    }
                }
            }
        }
        stage('Stage Ask Deploy') { 
            steps {
                input message: 'Deploy to production?'
            }
        }
        stage('Deployment to Production') { 
            agent any 
            stages {
                stage('Stage PRO.1') { 
                    steps {
                        echo "Stage PRO.1"
                        sleep(10)
                    }
                }
                stage('Stage PRO.2') { 
                    steps {
                        echo "Stage PRO.2"
                        sleep(10)
                    }
                }
            }
        }
    }
}