Reduce memory usage of ruby in RoR script

Question

I have a RoR app and a cron rake-task, something like:

Model.all.each do |m|
  if m < some_condition
    m.do_something
    m.save
  end
end

Model has 1 000 000 records (and 200 000 with acceptable conditions). Is there any way to improve task memory usage? It takes gigabytes of memory, and Ruby process is killed by the server on production. My DB is PostgreSQL.

Answer 1

You should use methods like #find_each and #find_in_batches. These will load only a small portion of records at a time. Take a look ActiveRecord::Batches.

Answer 2

I would suggest to use find_each, which yields your objects in batches.

Also, apply the condition you have inside the loop in sql if possible, so ActiveRecord does not have to instantiate the objects (and therefore use memory for) you're not using anyway:

Model.find_each(:conditions => {:my => :condition}).each do |m|
  # do something
end

Answer 3

you can try following method:

  def with_gc(enum)
    count = enum.count
    limit = 100
    (0..count).select{|i| i % limit == 0}.each do |index|
      new_count = enum.count
      raise "query depends on updated param. Expected count #{count}, got #{new_count}" if count != new_count
      enum.skip(index).limit(limit).each do |record|
        yield record
      end
      GC.start
    end
  end

you can use it like this:

with_gc(Model.all) do |m|
  if m < some_condition
    m.do_something
    m.save
  end
end