RuntimeError (Circular dependency detected while autoloading constant Apps multithreading

Question

I'm receiving this error: RuntimeError (Circular dependency detected while autoloading constant Apps

when I'm multithreading. Here is my code below. Why is this happening? The reason I am trying to multithread is because I am writing a HTML scraping app. The call to Nokogiri::HTML(open()) is a synchronous blocking call that takes 1 second to return, and I have 100,000+ pages to visit, so I am trying to run several threads to overcome this issue. Is there a better way of doing this?

class ToolsController < ApplicationController

def getWebsites
    t1=Thread.new{func1()}
    t2=Thread.new{func1()}
    t3=Thread.new{func1()}
    t4=Thread.new{func1()}
    t5=Thread.new{func1()}
    t6=Thread.new{func1()}
    t1.join
    t2.join
    t3.join
    t4.join
    t5.join
    t6.join
end

def func1
    puts Thread.current
    apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
    while apps.size == 1  do
        app = apps[0]
        puts app.name
        puts app.iTunes
        doc = Nokogiri::HTML(open(app.iTunes))
        array = doc.css('div.app-links a').map { |link| 
            url = link['href'] 
            url = Domainatrix.parse(url)
            url.domain + "." + url.public_suffix
        }
        array.uniq!
        if (array.size > 0)
            app.website = array.join(', ')
            puts app.website
        else
            app.website = "NONE"
        end
        app.save
        apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
    end 
end


end

Wow, can we start with what you're trying to do here? Firing off threads in a controller does not seem like a recipe for success. There are some SO answers that mention removing a :type field from your model if this happens, is that relevant? — Nick Veys
@etayluz, the code you've shared doesn't indicate what you want to do. Perhaps you don't need multithreading at all. If you want to simply get 6 random 'apps' objects, then do, for example t1, t2 = Apps.order("RANDOM()").where(conditions).limit(2).to_a with modifications for 6 instead of 2. — SHS
and if you must use multithreading, this might have something to do with "require" not being threadsafe. before your t1 variable's assignment. do t0 = Apps.first. it should require the necessary stuff before the threads are initiated. see if that helps — SHS
Thanks Nick and Syed, I have edited the question to answer your questions about the multithreading - is there a better way? — etayluz
The work you're trying to do here is probably better left to a background job, a la sidekiq. — lobati

SHS SHS · Accepted Answer · 2014-12-17T00:31:58

"require" isn't thread-safe

Change your methods so that everything that is to be "required" is done so before the threads start.

For example:

def get_websites
    # values = Apps.all # try uncommenting this line if a second-try is required

    ar = Apps.where("apps.website IS NULL")

    t1 = Thread.new{ func1(ar) }
    t2 = Thread.new{ func1(ar) }

    t1.join
    t2.join
end

def func1( ar )
    apps = ar.order("RANDOM()").limit(1)

    while (apps.size == 1)
      puts Thread.current
    end 
end

But as somebody pointed out, the way you're multithreading within the controller isn't advised.

RuntimeError (Circular dependency detected while autoloading constant Apps multithreading

1 Answers