Might be more of trying to overcome a learning curve + actual code question. I apologize if it seems nubish, currently I get this error within production.
Basically I keep recieving this "redis pool is too small" and I'm lost where to start, I'm actually lost on basically understanding how to accurately configure sidekiq with puma or anything that comes after configuration like scaling etc.
Below I have my configuration followed my error I recieve .
ProcFile web: bundle exec puma -C config/puma.rb worker: bundle exec sidekiq -e production -C config/sidekiq.yml
Sidekiq init
if Rails.env.production?
Sidekiq.configure_client do |config|
config.redis = { url: ENV['REDIS_URL'], size: 2 }
end
Sidekiq.configure_server do |config|
config.redis = { url: ENV['REDIS_URL'], size: 20 }
Rails.application.config.after_initialize do
Rails.logger.info("DB Connection Pool size for Sidekiq Server before disconnect is: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
ActiveRecord::Base.connection_pool.disconnect!
ActiveSupport.on_load(:active_record) do
config = Rails.application.config.database_configuration[Rails.env]
config['reaping_frequency'] = ENV['DATABASE_REAP_FREQ'] || 10 # seconds
# config['pool'] = ENV['WORKER_DB_POOL_SIZE'] || Sidekiq.options[:concurrency]
config['pool'] = 16
ActiveRecord::Base.establish_connection(config)
Rails.logger.info("DB Connection Pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
end
end
end
end
redis init
if ENV["REDISTOGO_URL"]
$redis = Redis.new(:url => ENV["REDISTOGO_URL"])
end
puma.rb
workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 1)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
# Because we are using preload_app, an instance of our app is created by master process (calling our initializers) and then memory space
# is forked. So we should close DB connection in the master process to avoid connection leaks.
# https://github.com/puma/puma/issues/303
# http://stackoverflow.com/questions/17903689/puma-cluster-configuration-on-heroku
# http://www.rubydoc.info/gems/puma/2.14.0/Puma%2FDSL%3Abefore_fork
# Dont have to worry about Sidekiq's connection to Redis because connections are only created when needed. As long as we are not
# queuing workers when rails is booting, there will be no redis connections to disconnect, so it should be fine.
before_fork do
puts "Puma master process about to fork. Closing existing Active record connections."
ActiveRecord::Base.connection.disconnect!
end
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
error in logs
Your Redis connection pool is too small for Sidekiq to work. Your pool has 20 connections but really needs to have at least 22
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/redis_connection.rb:38:in `verify_sizing'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/redis_connection.rb:17:in `create'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/lib/sidekiq.rb:128:in `redis='
10:04:44 worker.1 | /home/ubuntu/workspace/sample_v1/config/initializers/sidekiq.rb:10:in `block in <top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/lib/sidekiq.rb:70:in `configure_server'
10:04:44 worker.1 | /home/ubuntu/workspace/sample_v1/config/initializers/sidekiq.rb:9:in `<top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `block in load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/engine.rb:652:in `block in load_config_initializer'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/engine.rb:651:in `load_config_initializer'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `each'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `block in <class:Engine>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `each'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `tsort_each_child'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:415:in `call'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:415:in `each_strongly_connected_component_from'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:347:in `each'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:347:in `call'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
10:04:44 worker.1 | /usr/local/rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
10:04:44 worker.1 | /home/ubuntu/workspace/sample_v1/config/environment.rb:5:in `<top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/cli.rb:237:in `boot_system'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/cli.rb:50:in `run'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/sidekiq-4.1.4/bin/sidekiq:12:in `<top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/bin/sidekiq:23:in `load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/bin/sidekiq:23:in `<top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/cli/exec.rb:63:in `load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/cli/exec.rb:63:in `kernel_load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/cli/exec.rb:24:in `run'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/cli.rb:304:in `exec'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/cli.rb:11:in `start'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/exe/bundle:27:in `block in <top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/lib/bundler/friendly_errors.rb:98:in `with_friendly_errors'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/gems/bundler-1.12.3/exe/bundle:19:in `<top (required)>'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/bin/bundle:23:in `load'
10:04:44 worker.1 | /usr/local/rvm/gems/ruby-2.3.0/bin/bundle:23:in `<main>'
10:04:45 worker.1 | exited with code 1
10:04:45 system | sending SIGTERM to all processes
Any help or sort of guidance will be amazing.
