Releasing ActiveRecord connection before the end of a Sidekiq job

3
votes

This question deals with matters of performance and optimization with the way one can implement Sidekiq jobs.

Suppose we have the following job workflow

def perform

# Step 1
do_things_with_activerecord_db_and_get_some_parameters()

# Step 2
step2_perform_an_http_request_with_these_parameters_on_unreliable_server()

end

In the following case, an ActiveRecord connection is taken from the pool at Step 1, and will only be released by SideKiq after job has completed (or failed) by the ActiveRecord middleware of SideKiq.

Since the external http server at step2 on which we do the request is unreliable, the http request can take a long time or even timeout, and thus the ActiveRecord connection is locked for nothing for all that time, right?

So my question is: Is it pertinent, useful and safe to call:

ActiveRecord::Base.clear_active_connections!

between Step 1 and Step 2, so that the job frees the resource by itself and make it available for other similar jobs? Or have I missed something about connection pools? Can this method be applied to the redis connection too?

Thanks by advance!

1
ruby-on-railsrubyactiverecordconnectionsidekiq

1 Answers

4
votes

You definitely want to call clear_active_connections!.

We run ActiveRecord in an environment where we make use of JMS on a TorqueBox server, and we had to do something similar in order to ensure connections were freed.

As a caveat, you will also need to do this if you spawn a Thread which makes use of ActiveRecord, since (in ActiveRecord 3.2 certainly) the thread id is used as part of the connection checkout process.

A pattern we use repeatedly is the following:

def with_connection(&block)
  ActiveRecord::Base.connection_pool.with_connection do
    yield block
  end
ensure
  ActiveRecord::Base.clear_active_connections!
  ActiveRecord::Base.connection.close
end

which you could use like this:

with_connection do 
  do_things_with_activerecord_db_and_get_some_parameters()   
end