Stop Sidekiq gracefully while terminating Autoscale EC2 instance

Question

I am facing issue with aws autoscaling + sidekiq busy jobs. I have 4 instances are running in auto scale group and multiple Sidekiq processes are running on these instances. All instances using same redis.

If one of my instance is terminating at that time the jobs in his busy queue pushed into failed state. It should enqueued this jobs.

I have added one script in /etc/rc0.d folder which will kill sidekiq processes at the time of instace termination, but still my jobs are going in the failure state. I tried with TERM and USR1 signal for sidkeiq process termination but same thing happened with this signals.

I have used sidkeiq pro also enabled the reliable fetch.

Does anyone know, how to achieve the jobs in the busy queue should go into enqueued not in faliure state while killig sidekiq process manually or gracefully ?

Answer 1

I ran into the same issue myself and had to fix it by fully plumbing it into the daemon management, in my case chkconfig. I'm not sure what your EC2 setup is like, but I found that only scripts that chkconfig knew about were being executed.

If you don't already, you may also need to add a sleep to the script to ensure Sidekiq has enough time to fully exit.