Jenkins Pipeline and huge amount of parallel steps

Question

I have searched the whole internet for 2 weeks now, asked on freenode IRC and in the Jenkins user group mailing list for that but got no answer so here I am, you are my last hope (no pressure)

I have a Jenkins scripted pipeline that generates hundreds of parallel branches that have to run simultaneously on hundreds of slaves node. At the moment it looks like Jenkins BlueOcean user interface is not suited for that. We reach a point were all the steps can't be displayed.

I need to provide some kind of background to let you understand our need: We have a huge project in our company that have thousands of Behat/Selenium and this takes more that 30h to run now if done sequentially. We implemented a basic solution some times ago were we use a queuing system (RabbitMq) to store all the tests and consumers that run the tests by downloading the source code from Jenkins and uploading artifacts back to Jenkins too, but this is not as scallable as Jenkins native slaves and it is not maintainable enough (eg. we don't benefit from real time output log and usage statistics).

I know there is an open issue that describe the problem here : https://issues.jenkins-ci.org/browse/JENKINS-41205 but, basically, I need a workaround working for the next week (Our deelopment team are waiting for this new pipeline for a long time now).

Our pippeline looks like that at the moment:

Build --- Unit Tests --- Integration Tests --- Functional Tests ---
                |                |                    |
              tool A            suite A        matrix-A-A-batch 0
              tool B            suite B        matrix-A-A-batch 1
              tool C                           matrix-A-A-batch 2
                                               matrix-A-A-batch 3
                                                     ....
                                              "Unable to display more"

You can find a full version of our Jenkinsfile here : https://github.com/willy-ahva/pim-community-dev/blob/086e4ed48ef1a3d880ca16b6f5572f350d26eb03/Jenkinsfile (It may looks complicated but, basically, the real problem is the "Functional Tests" stage)

My questions are:

Am I using parallel the good way ?
Is it only a Jenkins/BlueOcean issue and I should contribute to the issue I linked ? (If yes, how ? I'm not a Java dev at all)
Should I try to use MultiJob and parallelize jobs instead of steps ?
Is there any other tool except parallel that I can use ? (some kind of fork or whatever) ?

Thanks a lot for your help. I love what Jenkins became with the Pipeline and BlueOcean UI and I really want to make it work in our team.

As Spencer Malone sugested we kept our queue system for this kind of parallelism. We changed a little bit our build so it is more easy to scale (by removing every many to one relation between the master and the consumers so it doesn't generate bottlenecks). — Clément Gautier
Did you try the original pipeline with hundreds of branches with new Jenkins versions? It seems the relevant bugs are fixed: issues.jenkins-ci.org/browse/JENKINS-41205 — Bash
Hey Bash, no I did not but you are right it's surely fixed now. We will keep our queue system because we rely on autoscaling that don't work well with native jenkins slave nodes but it's nice to know it's now possible. — Clément Gautier

Spencer Malone Spencer Malone · Accepted Answer · 2017-05-15T14:23:23

This is probably a poor way to do the parallel tasks. I would instead treat each parallel map entry as a worker, and put your tests into a queue / stack / data structure. Each worker thread could pop off the queue as required, and then you wouldn't sit there with a million tasks queued. You would have to be more careful with your logging so that it is apparent which test failed, but that shouldn't be too tough.
It's probably not something that's easy to fix, as it is as much a UI design issue as anything else. I would recommend that you give it a poke though! Who knows, maybe a solution will click for you?
Probably not. In my opinion this makes this muddier
Parallel is your option for forking.

If you really want to keep doing this, but don't want the UI to be so weird, you can stop defining each test as a stage. It'll be less clear what failed when one fails, but the UI should be happier.

Jenkins Pipeline and huge amount of parallel steps

1 Answers