Kubernetes Jenkins Plugin Creating more resources for pending slaves

Question

I have a question regarding how the Jenkins Kubernetes Plugin works. I am using google cloud, jenkins, helm, and kubernetes to set up my environment. I used these instructions to set up jenkins on kubernetes:

https://cloud.google.com/solutions/jenkins-on-kubernetes-engine-tutorial

In this tutorial through the helm file, the slaves consume 500mCPU and 256Mi memory with a limit of up to 1 CPU and 512MI. I then use the Kubernetes jenkins plugin which works how its supposed to I imagine. It basically uses my available resources in my kubernetes pools, or nodes, to create the agents to run my jobs on jenkins.

My problem is as follows, from the tutorial my available pools are 2 nodes with 2CPU's with 7.5GB each. Some of these resources are used by jenkins in order to have the server running etc which means I dont have all of the resources available. In my Jenkins server I changed the "Concurrency Limit" field to 20, I ran 20 jobs expecting for 20 executors to be created and used for my jobs at the same time but only 8 were created and used. The others were offline and suspended. I am guessing that only 8 were created because my cluster didnt have enough resources to create more executors? The executors were then able to run when some of the jobs finished and the executors got terminated. So now my question is, is there a way to add more resources (pods, nodes, or whatever it may be) in order for me to reach my concurrency limit if it happens to be that I am running 20 jobs so that my concurrency limit is reached and none of my jobs have to wait? I want to do this as some of my executors may not require a lot of CPU power(maybe like 100mCPU) but there may be others which may require 1CPU and with the cluster configurations in place now I will be able to run MAYBE 1 job at a time which will take long to execute all 20 jobs. If I am somehow able to resize my cluster by creating more nodes or pods then delete them, then that would make scaling easier while still being dynamic. Does anyone have any advice on how to achieve this? thank you

Answer 1

Since you're using Google Cloud, you can edit your existing node pool to enable autoscaling:

GKE's cluster autoscaler automatically resizes clusters based on the demands of the workloads you want to run. With autoscaling enabled, GKE automatically adds a new node to your cluster if you've created new Pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized and its Pods can be run on other nodes, GKE can delete the node.

This means that when you need, you'll have more nodes added to your node pool to met the resource demand for your workload.

Alternatively and if you have enough room in your node, you can increase the number of pods using Horizontal Pod Autoscaling:

Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization

This will create more pods instead of nodes, based on the pod resource usage metrics.