In our Rails application we use Sidekiq to run jobs that builds iOS apps.
The jobs call code that basically SSH onto a server and call a script to do various things such as generating files, exporting packages, and uploading apps to S3, etc.
At one point we had some issues with the server and the jobs are stuck under Busy (see screenshot).
We have tried multiple ways to try and remove these very dead jobs but they still exist.
Weirdly they don't even show up in the count on the queue:
queue = Sidekiq::Queue.new('ios')
queue.count
On both servers this returns 0 even though you can see them on the Web UI.
We have also tried:
require 'sidekiq/api'
Sidekiq::Queue.new('ios').clear
Sidekiq::Queue.new.clear
And the more aggressive full clearing of Redis:
Sidekiq.redis(&:flushdb)
But those Busy Jobs are still present...
We have even resorted to killing the threads themselves with:
# web3
Thread.list.map { |t| t.exit if t.object_id.to_s == 'xb69d' }
# web4
Thread.list.map { |t| t.exit if t.object_id.to_s == 'oxjjfzr2p' }
But those Jobs are still there in the UI... How can we kill them?
We're aware of: https://github.com/mperham/sidekiq/wiki/FAQ#how-do-i-cancel-a-sidekiq-job but these are jobs that have already been run and we can't now do anything in the application code to cancel them, so we need a solution to remove them alternatively.
