Sidekiq Unique Job Processing

Question

I need to assure that no more than one job per user_id is worked simultaneosly by any of the 25 workers to avoid deadlocks.

I have tried sidekiq unique jobs but deadlocks keep occoring because it keeps trying to process all pending jobs on the queue without looking for the user_id on the params.

Thank you

class MyWork
include Sidekiq::Worker
sidekiq_options :queue => "critical", unique: true, unique_args: ->(args) { [ args.user_id ] }

def perform(work_params)

Here is a not so great idea: You could probably maintain a table that contains the user ids that are being processed. Every time you have to start processing, take a look at the table and the current user id; start the processing only if the user id is not in the table. If it is in the table, push the job back to the queue. — bsvin33t
You should try to use the in memory store provided by redis to do this, instead of activerecord. — bsvin33t
I will investigate and possibly try that redis memory store if iy saves on resorces and provides data quicker, never heard about it @Vineeth. — ace

Bruno Peres Bruno Peres · Accepted Answer · 2016-06-13T12:15:40

The sidekiq_option args is an array that has your works_params on the first position. You should then do:

class MyWork
  include Sidekiq::Worker
  sidekiq_options :queue => "critical", unique: until_executed, 
                   unique_args: ->(args) { [ args.first.user_id ] }

  def perform(work_params)

Note that unique: true is deprecated, so you should use unique: until_executed or whatever fits best for your worker. See here for other options.

Sidekiq Unique Job Processing

1 Answers