deadlock when run two pipelines parallel in jenkins

Question

Currently I have the following problem. I have written a Jenkinsfile to Build my Repository in a Pipeline. Each Repository has their own Pipeline in a Multibranch Pipeline. Whenever I push the Repository the Pipeline start to Work.

For the building I have one Agent with two nodes. When the Multibranch Pipeline is running, the Multibranch Pipeline uses one node the execute a single Pipeline and the second node is used by the currently executing pipeline to run a single Job.

When two Pipelines run to the same time both Pipelines use one node. But now the Problem is both Pipelines can't start any Jobs since all nodes are occupied. At this time I have a deadlock since both pipelines are waiting for a free node for their jobs.

I have tried to set "disableConcurrentBuilds()", but this only blocks the Pipeline with the same name. Pipelines with different names in the Multibranch Pipeline can run concurrently.

A second try is to set Build Blocker Plugin with this code in the Jenkinsfile.

properties([
    [$class: 'BuildBlockerProperty',
     blockLevel: 'GLOBAL',
     blockingJobs: '*pipeline_Test*',
     scanQueueFor: 'ALL',
     useBuildBlocker: true],
   disableConcurrentBuilds()
   ])

But then I get this Error message.

WorkflowScript: 30: Invalid option type "properties". Valid option types: [buildDiscarder, catchError, checkoutToSubdirectory, disableConcurrentBuilds, disableResume, durabilityHint, lock, newContainerPerStage, overrideIndexTriggers, retry, script, skipDefaultCheckout, skipStagesAfterUnstable, timeout, waitUntil, withContext, withCredentials, withEnv, ws] @ line 30, column 4

How can I set the BuildBlockerProperty in the Jenkinsfile for the entire pipeline? Is there a other way to block all other pipelines so long the pipeline is running?

Thank you for help.

Answer 1

Struggled with the same issue.

You didn't give your Jenkinsfile but if your pipeline requires 1 executor node to run the pipeline and 1 additional to execute the job it is probably that you set an agent at pipeline level and also at stage level like

pipeline {
  agent any

  stage('Build') {
    agent {
      label 'my label'
    }

    steps {
      ...
    }
  }

  post {
    always {
      ...
    }
  }
}

as I did. Of course your specific agent setting can be very different, but the agent is set at 2 levels. The reason I had to specify a pipeline level agent in addition to the stage was because otherwise my post step wouldn't run.

If your pipeline requires specific agent settings (labels, docker images, ...) at stage level, it is best practice to set agent none at pipeline level:

pipeline {
  agent none

  stage('Build') {
    agent {
      label 'my label'
    }

    steps {
      ...
    }
  }
}

This way only 1 executor node is needed thus preventing the deadlock.

If like me you need a post step you can run it like

pipeline {
  agent none

  stage('Build') {
    agent {
      label 'my other label'
    }

    steps {
      ...
    }
  }

  post {
    always {
      node('label') {
        ...
      }
    }
  }
}

to provide it with a node. The label is mandatory, haven't found a way to run it on any node. Adding the node{} bloc allows the post block to run with agent none set at pipeline level.

This worked for me and might be a solution for the OP. Not enough information was given in the OP to know the specific configuration of the pipeline.

Answer 2

The easiest workaround to this is to just bump the number of executors by one, ensuring you always have one more executor than you do parent jobs.

This way there will always be either one executor free or one child job running which will complete and free the executor.

You can set executors in Manage Jenkins > Nodes > (node name) > Configure > # of executors

This isn't a full solution though, as you'd need to do it every time you add a job.

Answer 3

Use a Lock to lock a stage or step. This will prevent parallel building per branch in Multibranch pipelines.

stage("Do stuff") {
  lock("my_lock") {

    // do stuff

  } // resource is unlocked.
}

Answer 4

This is my solution, as I implemented it in the end.

The following picture shows my Windows nodes. The Pipeline must run on bildserver3 and the Jobs must run on buildserver2. Buldserver1 is the master.

And this is the jenkinsfile pipeline {

options {
    /*--- prevents the same branch from being built again at the same time ---*/
    disableConcurrentBuilds()
}

agent {
    node {
        /*--- define the pipeline to run on the buildserver3 ---*/
        label 'buildserver3'
    }
}


stages {
    stage('Set Build Name') {
        steps {
            script {
                 ...
            }
        }
    }
    stage('Checkout/Compile') {
        steps {
            node('buildserver2'){
                script {
                    ...
                }
            }
        }
    }
    stage('JUnit and Package') {
        steps {
            parallel(
                JUnit: {
                    node('buildserver2'){
                        script {
                            ...
                        }
                    }
                },
                Package: {
                    node('buildserver2'){
                        script {
                             ...
                        }
                    }
                }
            ) 
        }
    }
    stage('Install') {
        steps {
            node('buildserver2'){
                script {
                      ...
                }
            }
        }
    }
} 
post {
    success {
        node('buildserver2'){
          ...
        }
    }
    unstable{
        node('buildserver2'){
           ...
        }
    }
    changed {
        ....
    }
    always{
        ...
    }
}

}

It is important that the build server2 has as many nodes as jobs parallel are configured in jenkinsfile. Also, the number of parallel jobs must be multiplied by the number of nodes on buildserver3.

In this case, a maximum of two jobs are run in parallel, and build server3 has 3 accounts, so build server2 must have six accounts.

If a repository is created that has more parallel jobs, the number of nodes must be adjusted accordingly.

For a second slave on the same system, a second Jenkins service is required. How to configure this is described under the following link.

https://kimbouwman.wordpress.com/2014/07/14/multiple-slaves-on-same-machine-jenkins/