Sidekiq ignoring concurrency when using multiple queues

Question

I have two queues: development and ar_updater. I have a bunch of sidekiq workers that are continuously updating an ActiveRecord object, but sometimes they are querying the record at the same time, rather than only after another one has updated it. Therefore, I was thinking about just creating a queue and limiting its concurrency to 1 so that the correct data can be appended.

Here's my config/sidekiq.yml file:

development:  
  :concurrency: 10
ar_updater:
  :concurrency: 1
:queues:
  - development
  - ar_updater

and then I just have a simple worker that looks like this:

class ArUpdateWorker
  include Sidekiq::Worker
  sidekiq_options queue: "ar_updater"

  def perform(options)
    options = options.transform_keys(&:to_sym)
    schedule_id = options[:schedule_id]
    progress = options[:progress]

    schedule = Schedule.find(schedule_id)
    old_progress = schedule.current_progress
    schedule.update(current_progress: old_progress + progress)
  end
end

Is there a way to make sure that these workers are only running one at a time in this particular queue? When calling ArUpdateWorker multiple times, it seems like sidekiq ignores the concurrency and just runs the worker as many times as I want, but if I switch the queue of the worker to development, then it adheres to the concurrency of 10. A little confusing.

It seems the only way to workaround this is to run sidekiq with the concurrency settings set on the command line interface

TheGeorgeous TheGeorgeous · Accepted Answer · 2020-11-03T05:43:47

You cannot actually do that. You have a queue called development. That can be very confusing because you would also have an environment called development.

The configuration you have sets the configuration of the environment called "development" to 10, rather than the queue. A sidekiq process will have one concurrency setting depending on the environment. You can't set that process to have different concurrency levels.

This is a similar question: How can I specify different concurrency queues for sidekiq configuration?

Sidekiq ignoring concurrency when using multiple queues

1 Answers