Rails Resque workers fail with mysql "Too many connections"

Question

We recently switched our (ruby) job queueing system from DelayedJob to Resque.

While our latency has gone down, and we've eliminated the database bottleneck, we're now seeing a new problem; one or more of our workers seems to leave a database connection open when it exits. When we look at the process list, there are hundreds of connections in a 'sleep' state. They eventually time out after 90 seconds. We've been throttling back our workers to keep from running out of client connections, but what we really need to find out is which one (or more) of our jobs is not being polite when it disconnects using the mysql2 ruby client.

Any ideas how we could (1) find the culprits or (2) instrument our code so we can make sure that we are actually disconnecting before the job terminates?

Rails 4.0.x
Resque 1.25.2
mysql2 gem 0.3.16

Seeing the same problem on our Rails 4.2.6 application with the pg gem v0.18.4. — Ben

infused infused · Accepted Answer · 2014-07-16T22:03:04

Make sure your Resque process is disconnecting from the database before forking and re-establishing the connection afterwards. Create an initializer file config/initializers/resque.rb which contains:

Resque.before_fork do
  defined?(ActiveRecord::Base) && ActiveRecord::Base.connection.disconnect!
end

Resque.after_fork do
  defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
end

Rails Resque workers fail with mysql "Too many connections"

2 Answers