Sidekiq stopped processing jobs in queue

Question

Sidekiq had been processing jobs just fine (finished 30 jobs overnight). This morning, it completely shut off processing the queue (now up to 28 jobs).

Running on Heroku (1 Standard-2x Web Dyno, 1 Standard-1x Worker Dyno).

Procfile (where I have had the most trouble finding documentation to configure)

web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e production -C config/sidekiq.yml

sidekiq.yml

development:
 :concurrency: 5
production:
 :concurrency: 20
:queues:
 - ["default", 1]
 - ["mailers", 2]

sidekiq.rb

if Rails.env.production?

  Sidekiq.configure_client do |config|
    config.redis = { url: ENV['REDIS_URL'], size: 2 }
  end

  Sidekiq.configure_server do |config|
    config.redis = { url: ENV['REDIS_URL'], size: 20 }

    Rails.application.config.after_initialize do
      Rails.logger.info("DB Connection Pool size for Sidekiq Server before disconnect is: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
      ActiveRecord::Base.connection_pool.disconnect!

      ActiveSupport.on_load(:active_record) do
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DATABASE_REAP_FREQ'] || 10 # seconds
        # config['pool'] = ENV['WORKER_DB_POOL_SIZE'] || Sidekiq.options[:concurrency]
        config['pool'] = 16
        ActiveRecord::Base.establish_connection(config)

        Rails.logger.info("DB Connection Pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
      end
    end
  end

end

Also, with all the jobs in the queue (default, mailers) is it possible to have Sidekiq force run the jobs?

UPDATE

I have narrowed the error to the Heroku worker. Upon restart, the worker quickly crashes.

The first error had to do with Sidekiq not spawning enough connections (Redis required 22, I had set the limit to 20).

Now, I am getting the following error:

Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

I am running PG Hobby via Heroku. Its connection limit is 20. Is this the source of the issue?

About "Is it possible to have Sidekiq force run the jobs?", yes are able to check the queues and what's being processed on Sidekiq monitor: github.com/mperham/sidekiq/wiki/Monitoring — AndreDurao
@AndreDurao monitoring only gives the option to delete a queued job. For example, in the mailers, I would like to force-run the job. — jcxswc
Did you tried to restart the worker? devcenter.heroku.com/articles/… — AndreDurao
@AndreDurao see the above update. Turns out the worker has been crashing frequently. — jcxswc

jcxswc jcxswc · Accepted Answer · 2018-04-19T15:02:17

After 2 days on this venture, answer discovered thanks to this SO question. Hopefully, this helps the SEO for future onlookers.

I changed my sidekiq.rb file to

sidekiq.rb

require 'sidekiq/web'

Sidekiq.configure_server do |config|
  ActiveRecord::Base.configurations[Rails.env.to_s]['pool'] = 30
end

if Rails.env.production?
  Sidekiq.configure_server do |config|
    config.redis = { url: ENV["REDISTOGO_URL"]}
  end
  Sidekiq.configure_client do |config|
   config.redis = { url: ENV["REDISTOGO_URL"]}
  end
end

Then, I updated my sidekiq.yml to

sidekiq.yml

queues:
- default
- mailers

:verbose: false
:concurrency: 5 # Important to set depending on your Redis provider
:timeout: 8 # Set timeout to 8 on Heroku, longer if you manage your own systems.

Note: There is also the option to leave out everything in the if statement in the initializer file. As stated by the creator of Sidekiq, you can set your heroku config to:

heroku config:set REDIS_PROVIDER=REDISTOGO_URL

and Sidekiq will figure out the rest.

Sidekiq stopped processing jobs in queue

1 Answers