Sidekiq multiple dependant jobs, when to completed or retry?

Question

In my Rails application, I have a model called Report

Report has one or many chunks (called Chunk) that would generate a piece of content based on external service calls (APIs, etc.)

When user requests to generate a report, by using Sidekiq, I queue the "chunk's jobs" in order to run them in the background and notify user that we will be emailing them the result once the report is generated.

Report uses a state machine, to flag whether or not all the jobs are successfully finished. All the chunks must be completed before we flag the report as ready. If one fails, we need to either try again, or give up at some point.

I determined the states as draft (default), working, finished The finish result is a combination of all the services pieces together. 'Draft' is when the chunks are still in the queue and none of them has started generating any content.

How would you tackle this situation with Sidekiq? How do you keep a track (live) which chunk's services are finished, or working or failed, so we can flag the report finished or failed?

I'd like to see a way to periodically check the jobs to see where they are standing, and change a state when they all finished successfully, or flag it fail, if all the retries give up!

Thank you

scilence scilence · Accepted Answer · 2020-06-06T21:00:43

We had a similar need in our application to determine when sidekiq jobs were finished during automated testing.

What we used is the sidekiq-status gem: https://github.com/utgarda/sidekiq-status

Here's the rough usage:

job_id = Job.perform_async()

You'd then pass the job ID to the place where it will try to check the status of the job

Sidekiq::Status::status job_id #=> :working, :queued, :failed, :complete

Hope this helps.

Sidekiq multiple dependant jobs, when to completed or retry?

2 Answers