After reading this blog: https://manuelvanrijn.nl/blog/2012/11/13/sidekiq-on-heroku-with-redistogo-nano/
I noticed SideKiq had two settings that I originally thought referred to the same thing:
Concurrency is set in the sidekiq.yml and server pool size is set in the sidekiq.rb file in the initializer folder.
What is the difference between these two settings?
Concurrency - number of tasks that can be performed parallelly(threads) in sidekiq.
If concurrency is 2 and 10 jobs come to sidekiq, it will execute 2 jobs at a time. Once the first two jobs are completed then only it can execute the next jobs.
Server Pool Size - In order to fetch jobs from Redis sidekiq will have to connect to the Redis server. A Redis server will have a limit of max connections. Server pool size will limit the max number of connections a sidekiq thread can open to the Redis server. A single thread will at max need 3 Redis connection. So if the concurrency is 2 then the server pool size should be 6.
If we don't add server pool size, sidekiq will create new connections to Redis as needed(instead of reusing the existing open connections) this might exhaust the number of connections available in the Redis
Concurrency is the number of job execution threads and makes a big difference in how much RAM your process will consume during its life. Pool size is the number of Redis connections in the process. A Sidekiq process must have at least (concurrency + 2) connections available.
I strongly recommend you don't configure the pool size at all, let Sidekiq do it for you.
