How to finish long-running task on Tomcat when AWS AutoScaling is terminating the EC2 instance?

Question

I have an application that is deployed to Tomcat 8 that is hosted on ElasticBeanstalk environment with enabled auto-scaling. In the application I have long-running jobs that must be finished and all changes must be committed to a database.

The problem is that AWS might kill any EC2 instance during scale in and then some jobs might be not finished as it is expected. By default, AWS waits just 30 seconds and then kill the Tomcat process.

I've already changed /etc/tomcat8/tomcat8.conf file: set parameter SHUTDOWN_WAIT to 3600 (60 by default). But it didn't fix the issue - the whole instance is killed after 20-25 minutes.

Then I've tried to configure lifecycle hook via .ebextensions file (as it's explained here). But I couldn't approve that the lifecycle hook really postpones termination of the instance (still waiting for an answer from AWS support about that).

So the question is: do you know any "legal" ways to postpone or cancel instance termination when the autoscaling group scales in?

I want to have something like that:

• AWS starts to scale in the autoscaling group
• autoscaling group sends shutdown signal to the EC2 instance
• EC2 instance starts to stop all active processes
• Tomcat process receives a signal to shutdown, but waits until the active job is finished
• the application commits the job result (it might take even 60 minutes)
• Tomcat process is terminated
• EC2 instances in terminated

Answer 1

Elastic Beanstalk consists of two parts - API, and Worker. API is auto scaled, so it can go down. Worker is something that runs longer. You can communicate between them with SQS. That is how they designed it.

For sure you can tweak the system. That is platform as a service, so you can force auto scaling group not to go down - by setting min instances to max. Also you can turn off health check - that can also kill instance... But that is hacking, latter can kick.