How to launch multiple AWS EC2 machines on Jenkins?

4
votes

I am using the EC2 plugin in Jenkins (https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin) for using AWS services.

As of now, a job which uses this cloud label spins up a single EC2 instance. Whereas, I need to spin multiple instances for this job.

A single EC2 instance will never be overloaded and hence the plugin will not spin any new EC2 instances; but the job needs multiple instances to distribute the tasks. There are ~100 tasks to be assigned to 20 machines one at a time (hence 5 per machine). A single instance can complete one task at a time.

How can I spin multiple EC2 instances (fixed number say 20) for a single job in Jenkins?

Thanks.

1
amazon-web-servicesjenkinsamazon-s3amazon-ec2jenkins-plugins
Try increasing the instance cap under the advanced section for the instance configuration. – Venkata.Mutyala
Hi Venkata. I have already configured this. Even if the instance cap value is not set; the default value for number of EC2 instances in 20. Is there a way in which I can spin say 20 machines right when the jib starts( since the job will launch one instance after another, that too if the load on the first instance increases, which will not increase in my case). – vintrojan
I have a feeling you already did this.... but could you confirm if the instance executors has been capped to 1 and if you have the Jenkins project set to allow concurrent runs? – Venkata.Mutyala
@Venkata.Mutyala Thanks for the input. The job spins multiple instances, but the delay is too much; since the plugin checks for cluster overload and then starts a new instance; which is almost 90 seconds. I want the job to launch EC2 instances more frequently (~1-2 seconds). Since, to spin 100 instances it will take 90 seconds *100 = 9000 seconds (~2.5 hrs). – vintrojan

1 Answers

2
votes

You can try the spot fleet plugin instead which has more scaling options

https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Fleet+Plugin