Tracking run time of various asynchronous background jobs in Sidekiq

0
votes

Let's say I'm pushing several jobs on to a sidekiq queue to be executed immediately and asynchronously. I have log messages that tell me when the overall process has started and completed

start = Time.zone.now
puts "Starting jobs"

1000.times.each do
  SomeWorker.perform_async(some_arg_id)
end

end_time = Time.zone.now
runtime = end_time - start
puts "Finished jobs - that took #{runtime} seconds"

This of course has one very big flaw - it outputs that the jobs have completed when everything is pushed to the queue, not when the jobs have actually completed.

I'm looking for a way to track the completion of a batch of jobs. I'd hate to keep polling every X seconds to see if the queue is empty. It also seems hacky to have the last job in the queue update the DB or log with the completed time, because they're not guranteed to finish in any order.

Is there a best practice for approaching this?

Thanks!

1
ruby-on-railsasynchronoussidekiq

1 Answers

2
votes

Well, the best approach seems to use Sidekiq Pro: https://github.com/mperham/sidekiq/wiki/Batches

You can fetch the status of a Batch programmatically.

I can't see any other way apart from using pro or polling the queue size manually.