Jenkins pipeline taking up “mirrored” executors on master

Question

I am porting over a built to use Jenkins' declarative pipeline. Everything is going great, but I am seeing something odd that I didn't expect in my currently running pipeline. The pipeline is setup to do all stages and steps on remote agent systems. The pipeline has parallel stages that spawn off quite a bit of work across 6-8 executors across 3 agents. This all works great.

But for some reason, while the pipeline is executing there is an executor consumed on the master node that seems to "mirror" one of the running remote executors. It doesn't appear to be doing any work (very low CPU usage on the master) but it is definitely taking an executor slot and has the name at the end (ex: "#74 - (tests)") that matches the name of one of the currently executing stages on another agent.

At first I thought this may be a "fly-weight" running on the master, but from what I have found online it sounds like those should not take an executor.

It wouldn't be much of a problem except, when multiple builds are running from multiple branches, I run out of executors on the master node and then builds start to pile up. For now I have just allocated a large number of executors on master, but I would rather not do that since it will also build up a large amount of storage space used for workspaces for all the executors.

My question is: What are these executors and is this to be expected?

MaratC MaratC · Accepted Answer · 2020-01-05T08:07:46

From what we see, this is the thing that watches over parallel part of your pipeline. In your example, #74 - (tests) is probably a parallel stage.

This stage does not take any additional storage over the storage that is already allocated, so its storage requirements should not bother you.

You can also run your pipeline on a dedicated node with a label, like this:

pipeline {
    agent {
        node {
            label 'master_pipeline_builder'
        }
    }

Allocate quite an amount of executors on that node, and this should solve your issues without affecting your master node.

Jenkins pipeline taking up “mirrored” executors on master

1 Answers