Calling worker inside another worker with sidekiq,

Question

I have problem with calling worker inside another worker with sidekiq. I have a two workers that looks like this:

class UserExportWorker
  include Sidekiq::Worker

  sidekiq_options queue: :users, unique: true, retry: false

  def perform(user_id)
    # code that exports user to external service via API
    UserExportStatusWorker.perform_in(1.minute, user_export_id, user_id)
  end
end

class UserExportStatusWorker
  include Sidekiq::Worker

  sidekiq_options queue: :users, unique: true, retry: false

  def perform(user_export_id, user_id)
    # code that check if exporting user to external service via API is finished
    if export.completed?
      user.update(status: 'exported')
    else
      UserExportStatusWorker.perform_in(1.minute, user_export_id, user_id)
    end
  end
end

UserExportWorker exports user to some external service via API and run UserExportStatusWorker that will check if export in external API is finished and, if its not then it will launch the same worker again, when export will finish it will update user state column in database. The problem is that UserExportStatusWorker is not updating user in database even if export to external API is successfull. But when I change my code to this:

class UserExportWorker
  include Sidekiq::Worker

  sidekiq_options queue: :users, unique: true, retry: false

  def perform(user_id)
    # code that exports user to external service via API
    UserExportStatusWorker.new.perform(user_export_id, user_id)
  end
end

class UserExportStatusWorker
  include Sidekiq::Worker

  sidekiq_options queue: :users, unique: true, retry: false

  def perform(user_export_id, user_id)
    # code that check if exporting user to external service via API is finished
    if export.completed?
      user.update(status: 'exported')
    else
      UserExportStatusWorker.new.perform(user_export_id, user_id)
    end
  end
end

I'm completely out of ideas... Why this not works?

Yes, second example works properly... No they not. In second example I use new.perform instead of perfrom_in — Mateusz Urbański
are you certain jobs in UserExportStatusWorker are getting enqueued with perform_in ? — Anthony
In sidekiq I have visual UI that gives me information about current jobs. When I use perform_in I see in sidekiq admin panel that job is scheduled in 1 minute. But then notting happen. Job dissappear from scheduled jobs but counter of successfull finished jobs didn't change — Mateusz Urbański
If you're not using an error handling service, I suggest you temporarily set retry: true and watch the retry queue in the UI to see if the job is failing. I use rollbar but he recommends a few - github.com/mperham/sidekiq/wiki/Error-Handling — Anthony
@Anthony, when I change unique to false in UserExportStatusWorker, everything works as expected. — Mateusz Urbański

Bruno Peres Bruno Peres · Accepted Answer · 2017-04-20T14:43:47

Ideally to cause a restart on a Worker you should be raising an Error, and letting sidekiq retry it, as described in the docs.

So in your case, in your UserExportWorker class you could have a method to check if the exporting ran successfully, and if it didn't, raise a custom ExportNotSuccessfulError

Calling worker inside another worker with sidekiq,

1 Answers