Jenkins AWS Spot fleet plugin doesn't automatically scale spot instances

6
votes

Planned to use EC2 Spot instance/fleet as our jenkins slave solution based on this article https://jenkins.io/blog/2016/06/10/save-costs-with-ec2-spot-fleet/.

EXCEPTED
if the spot instances nodes remain free for the specified idle time (I have configured for 5 minutes), then Jenkins releases the nodes, and my Spot fleet nodes will be automatically scaled down.

ACTUAL
my spot instances is still running for days.Also, noticed when I have more pending jobs, Jenkins does not automatically scale my Spot fleet to add more nodes.

Automatic scale up/down supposed to be triggered automatically by aws service? or is this supposed to be triggered by the jenkins plugin?

CONFIGURATION
Jenkins version : 2.121.2-1.1
EC2 Fleet Jenkins Plugin version : 1.1.7
Spot instance configuration :

  • Request type : request & maintain
  • Target Capacity : 1

Spot fleet plugin configuration :

  • Max Idle Minutes Before Scaledown : 5
  • Minimum Cluster Size : 0
  • Maximum Cluster Size : 3

Any help or lead would be really appreciated.

1
amazon-web-servicesjenkinsamazon-ec2
hmm the plugin might be broken, best bet will be add a post trigger shutdown call after the job runs. – Banjo Obayomi
I suggest you try the latest version of plugin 1.1.9 I see a lot of commits around scale down and termination. – terma
Scale down/up is done by Plugin, EC2 Fleet responsibility to maintain the required amount of nodes. – terma
@Faisal Was there a solution to this? – Chris F

1 Answers

3
votes

I had the same issue and by looking in Jenkins' logs I saw it tried to terminate the instances but was refused to by AWS.

So, I checked in AWS Cloudtrail all the actions Jenkins tried and for which there was an error.

In order for the plugin to scale your Spot Fleet, check that your AWS EC2 Spot Fleet plugin has the following permissions with the right conditions:

ec2:TerminateInstances
ec2:ModifySpotFleetRequest

In my case, the condition in the policy was malformed and didn't work.