My autoscaling group in AWS is all messed up... I tried scaling down, but for some reason it wasn't terminating the instances... they just stayed stuck at "Terminating:Wait" for hours. So I gave up and just terminated them manually, but now they're still there in the Auto Scaling group's Instance list, saying "Terminating:Wait" even though they no longer exist in any way.
How do I fix this?
I had the same problem. I wanted to destroy my Auto Scaling Group, but the lifecycle hook was causing issues. Specifically, the status of the instance (as listed under instances on the ASG console) was terminating:wait while the instance was marked as terminated on the regular EC2 instance console.
After some Googling, here's what I figured out. If you'd like to manually "ABANDON" the lifecycle hook on the instance that's causing you an issue, here's how you do that (with the AWS CLI):
aws autoscaling complete-lifecycle-action \
--lifecycle-hook-name ${LIFECYCLE_HOOK_NAME} \
--auto-scaling-group-name ${ASG_NAME} \
--lifecycle-action-result ABANDON \
--instance-id ${EC2_INSTANCE_ID}
https://docs.aws.amazon.com/cli/latest/reference/autoscaling/complete-lifecycle-action.html
It appears that you have Lifecycle Hooks activated on your Auto Scaling group.
When an instance is terminated with a Lifecycle Hook active, the instance is not immediately terminated. Rather, it is placed into the Terminating:Wait state and a message is sent via SNS or SQS to signal the pending termination. You are meant to have a process that receives this message, does whatever it needs to do (eg copy off log files) and then signals that the instance may be terminated.
If the termination signal is not received after 90 minutes, the instance will be terminated. So, you'll probably find they are gone now.
It's very strange to have Lifecycle Hooks activated unintentionally because they can only be added via an API call.
