What is the recommended way of load balancing a multi-master/multi-slave Jenkins setup

Question

I would like to setup an N-master x M-slave Jenkins build/execution farm.

I'd like to allow multiple projects to run their own Jenkins master server, each distributing their builds into the same pool of Jenkins slaves.

So each slave will have separately configured workspaces and so on.

What is the recommended way of load balancing the jobs across the slaves? i.e. if Master 1 has kicked off a job on Slave 1, and Master 2 comes along to run a job, I'd like it to pick a slave other than 1 to run that job on.

What is the recommended way of setting that up?

Answer 1

There are couple of ways you could do this. You could create the job to run on any open executor. An executor is what Jenkins runs jobs on. Assuming each instance of Jenkins has one executor, if one master executes something on Slave1 it will then take up one executor. So when master two tries to run something, it will run it on Slave2 (or any other executor that is available), since Slave1 is busy.

This depends on two things:

1. that all jobs can run on any available machine/executor
2. that each Jenkins has one executor.

Jenkins does manage load as best as it can, but I would recommend having one executor for every core available on the machine, so your machines aren't flogged down, and let Jenkins handle where they are actually run. You can even tie jobs to specific machines as well.

Good luck.