Tell Sidekiq to use all available Heroku workers

2
votes

I need to batch process a large set of files (millions of database records) as quickly as possible. For this purpose, I split the files into 3 directories and set up Sidekiq with the standard configuration (no config file).

I then started 3 Heroku workers and called 3 methods, which started 3 Sidekiq workers, all with the "default" queue. Initially, Sidekiq used 2 Heroku workers and after a while it decided to use only 1 worker.

How can I force Sidekiq to use all 3 workers to get the job done asap?

Thanks

enter image description here

1
ruby-on-railsherokusidekiq

1 Answers

6
votes

I found the solution at the bottom of this page: http://manuelvanrijn.nl/blog/2012/11/13/sidekiq-on-heroku-with-redistogo-nano/

# app/config/sidekiq.yml
:concurrency: 1

# Procfile    
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -C config/sidekiq.yml

Also, if you have many workers and a free / cheap Redis instance, make sure you limit the number of connections from each worker to the Redis server:

# app/config/initializers/sidekiq.rb
require 'sidekiq'

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

Sidekiq.configure_server do |config|
  config.redis = { :size => 2 }
end

You can calculate the maximum of connections here: http://manuelvanrijn.nl/sidekiq-heroku-redis-calc/