Jenkins EC2 plugin: instances not starting

Question

This is my Jenkins EC2 configuration:

URL: $JENKINS_URL/configureClouds/
Add new cloud: Amazon EC2
Name: Amazon EC2 eu-central-1
Amazon EC2 Credentials: AKIA...
Region: eu-central-1
EC2 Key Pair's Private Key: ubuntu
Test connection: success
Advanced...
Instance Cap: 3
No delay provisioning: checked
Add AMI
Description: Linux node
AMI ID: ami-0293...
Check AMI: 05052029...
Instance Type: T3aMedium
EBS Optimized: checked
Monitoring: checked
T2 Unlimited: checked
Security group names: sg-0c2d... (opens SSH port 22)
Remote FS root: ./jenkins
Remote user: ubuntu
AMI Type: unix
Labels: aws ubuntu linux
Usage: Use this node as much as possible
Idle termination time: 30
Advanced...
Number of executors: 2
Stop/Disconnect on Idle Timeout: checked
Minimum number of instances: 1
Minimum number of spare instances: 0
Instance cap: 10
Block device mapping: /dev/sda1=snap-0eadbe3f...:200:true:gp2, /dev/sdb=ephemeral0, /dev/sdc=ephemeral1
Associate Public IP: checked
Connection Strategy: Public DNS
Host Key Verification Strategy: off
Maximum Total Uses: 10
Environment variables: checked (not listing all environment variables)
Tool locations: checked (not listing all tool locations)

With this configuration, I would expect that at least 1 EC2 instance would be started, but no instance is started.

In the nodes page in Jenkins when I hit the provision via button, I get an error:

Oops! A problem occurred while processing the request. Logging ID=8ead3651-3809-4a47-984c-e0e494c705bb

In /log/all I have:

Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate getImage
Getting image for request {ExecutableUsers: [],Filters: [],ImageIds: [ami-0293c4ed***],Owners: []}
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{description='Linux node', labels='aws ubuntu linux'}. Considering launching
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
AMI had /dev/sda1
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice
{DeleteOnTermination: true,SnapshotId: snap-0eadbe3f***,VolumeSize: 20,VolumeType: gp2,Encrypted: false}
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{description='Linux node', labels='aws ubuntu linux'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Apr 14, 2021 5:34:37 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
SlaveTemplate{description='Linux node', labels='aws ubuntu linux'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-0293c4ed***]}, {Name: instance-type,Values: [t3a.medium]}, {Name: key-name,Values: [***]}, {Name: tag:jenkins_server_url,Values: [https://jenkins.***.com/]}, {Name: tag:jenkins_slave_type,Values: [demand_Linux node]}],InstanceIds: [],}
Apr 14, 2021 5:34:37 PM WARNING hudson.init.impl.InstallUncaughtExceptionHandler handleException
Caught unhandled exception with ID c080ae42-6b7b-47aa-93ea-1b8064503c1c
com.amazonaws.services.ec2.model.AmazonEC2Exception: Value () for parameter groupId is invalid. The value cannot be empty (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: da74dbbf-0685-45ac-8454-c3f5d1b4c700; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:29240)
    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:29207)
    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:29196)
    at com.amazonaws.services.ec2.AmazonEC2Client.executeRunInstances(AmazonEC2Client.java:28011)
    at com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:27980)
    at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:1100)
    at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:1042)
    at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:867)
    at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:693)
    at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:430)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
    at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
    at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at org.jenkinsci.plugins.modernstatus.ModernStatusFilter.doFilter(ModernStatusFilter.java:50)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
    at java.lang.Thread.run(Thread.java:748)

What does the instance log in Jenkins say when you try to start an instance? Does it have the iam permissions required to start an instance — apr_1985
@apr_1985 there can only be an instance log when an instance starts. I don't even get that far. Also, and I don't know if you know this, with the EC2 plugin you cannot "just" start an instance. The plugin automatically creates and destroys instances, and when there is zero load on Jenkins, then the instances do not exist at all. However I have configured to have at least one instance online at all times, and even that one isn't created. — Amedee Van Gasse
And it's the root user so it definitely has permissions. Bad practice, I know, but as long as IT doesn't tell me to do it otherwise, then it just works. — Amedee Van Gasse
I meant in the nodes page in Jenkins when you hit the provision via <cloud Name> button what errors are you seeing (rather than the AWS instance log). Also root user is that an AWS admin user as that is shocking practice :D — apr_1985
I get "Oops! A problem occurred while processing the request. Logging ID=8ead3651-3809-4a47-984c-e0e494c705bb" — Amedee Van Gasse

apr_1985 apr_1985 · Accepted Answer · 2021-04-15T08:11:20

Write up of the comments for anyone else looking for help diagnosing EC2 Agent Plugin issue.

When you have configured your agents go to the Nodes page (Jenkins URL/computer)
Hit the button to Provision a new agent from your cloud
If there is a configuration issue you will get Evil Jenkins and a Logging ID
Go to Jenkins logs page (Jenkins URL/log/all) and search that ID
This should give you the stack trace from the AWS SDK call which will help you to narrow down whether is it missing config or IAM permissions etc at fault

If there were no config errors you would get taken to the node that is being launched config page where you would be able to see its EC2 startup log to check for any User Data or AMI issues.

Jenkins EC2 plugin: instances not starting

1 Answers