Application not running in free heroku account

0
votes

I have a free heroku account where I have deployed my rails app. I have also enabled the redis heroku addon. My app scrapes sites and in my local env it took 93 mins to complete. So i decided to put it in background task and run it there. I followed some tutorials to get background task running in free heroku account so I used unicorn server. While testing it locally my app works fine but once I deploy it to heroku and browse my project site it does not work.Here is the result from heroku log

2016-09-16T17:40:40.934711+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.943447+00:00 app[web.1]: E, [2016-09-16T17:40:40.943337 #3] ERROR -- : reaped #<Process::Status: pid 13987 exit 1> worker=2
2016-09-16T17:40:40.951820+00:00 app[web.1]: E, [2016-09-16T17:40:40.946468 #13992] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.951822+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.951825+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.951827+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.951828+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.951829+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.961433+00:00 app[web.1]: E, [2016-09-16T17:40:40.961347 #3] ERROR -- : reaped #<Process::Status: pid 13992 exit 1> worker=1
2016-09-16T17:40:40.966761+00:00 app[web.1]: E, [2016-09-16T17:40:40.953957 #13996] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.966764+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.966765+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.966766+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.966768+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.966769+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.966770+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.966771+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.968861+00:00 app[web.1]: E, [2016-09-16T17:40:40.944646 #13990] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.968863+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.968864+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.968865+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.968866+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.968867+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.968872+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.968871+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977007+00:00 app[web.1]: E, [2016-09-16T17:40:40.976925 #3] ERROR -- : reaped #<Process::Status: pid 13990 exit 1> worker=0
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.977910+00:00 app[web.1]: E, [2016-09-16T17:40:40.965528 #13999] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.977913+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.977915+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.977916+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.977917+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977919+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.981913+00:00 app[web.1]: E, [2016-09-16T17:40:40.981796 #3] ERROR -- : reaped #<Process::Status: pid 13996 exit 1> worker=2
2016-09-16T17:40:40.982020+00:00 app[web.1]: E, [2016-09-16T17:40:40.981968 #3] ERROR -- : reaped #<Process::Status: pid 13999 exit 1> worker=1
2016-09-16T17:40:40.998699+00:00 app[web.1]: E, [2016-09-16T17:40:40.985434 #14004] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.998703+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.998704+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.998705+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.998706+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.998707+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.998709+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.998710+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.013015+00:00 app[web.1]: E, [2016-09-16T17:40:41.012891 #3] ERROR -- : reaped #<Process::Status: pid 14004 exit 1> worker=0
2016-09-16T17:40:41.013269+00:00 app[web.1]: E, [2016-09-16T17:40:40.992746 #14009] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.013271+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.013273+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.013274+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.013275+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.013277+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.015082+00:00 app[web.1]: E, [2016-09-16T17:40:40.991259 #14006] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.015083+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.015084+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.015085+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.015087+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.015089+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.029456+00:00 app[web.1]: E, [2016-09-16T17:40:41.019600 #14013] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.029459+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.029465+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.029466+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.033517+00:00 app[web.1]: E, [2016-09-16T17:40:41.032905 #3] ERROR -- : reaped #<Process::Status: pid 14006 exit 1> worker=1
2016-09-16T17:40:41.033606+00:00 app[web.1]: E, [2016-09-16T17:40:41.033566 #3] ERROR -- : reaped #<Process::Status: pid 14009 exit 1> worker=2
2016-09-16T17:40:41.034410+00:00 app[web.1]: E, [2016-09-16T17:40:41.034370 #3] ERROR -- : reaped #<Process::Status: pid 14013 exit 1> worker=0
2016-09-16T17:40:41.067342+00:00 app[web.1]: E, [2016-09-16T17:40:41.057962 #14018] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.067346+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.067348+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.067350+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.067351+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.067352+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.077263+00:00 app[web.1]: E, [2016-09-16T17:40:41.077156 #3] ERROR -- : reaped #<Process::Status: pid 14018 exit 1> worker=1

Here is my unicorn config

unicorn.rb

before_fork do |server, worker|
   @sidekiq_pid ||= spawn("bundle exec sidekiq -c 2")
end

worker_processes 3
after_fork do |server, worker|
  Sidekiq.configure_client do |config|
    config.redis = { :size => 1 }
  end
  Sidekiq.configure_server do |config|
    config.redis = { :size => 5 }
  end
end

Procfile

web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb

1
ruby-on-railsherokusidekiqunicorn
I would look into using Heroku's task scheduler and setting the background scraper up to run in a rake task (much simpler and easier than using sidekiq). Then just using the proper Procfile you'll be good to go. Also, with that you won't need RedisToGo on Heroku since their scheduler will execute it via rake. Let me know if you want a full write up on how to go that route and I'll post an answer. – bkunzi01
Hi @bkunzi01 , after reading the docs on Herokus task scheduler I found out Anything that takes longer than a couple of minutes to complete should use a worker dyno to run.` about this. My scraper takes 93 mins to run completely so will task scheduler help me with this? – Raaz
@bkunzi01 , hi, if you do not mind, can you post the full write up plz. – Raaz
No proboem. Answer incoming – bkunzi01

1 Answers

1
votes

Firstly, setup your procfile to properly utilize the worker dynos (im using puma for web so change that to reflect your unicorn setup):

web: bundle exec puma -C config/puma.rb
worker: bundle exec rake jobs:work

Now, to create a rake task is very simple. Open up your lib/tasks folder and add a file called "scraper.rake". This is where you're going to put the code that runs your scraper. Since you didn't show your scraper code I added some of mine so you have an example. First, require any outside files or configs youll need for running the scraper. Then you wrap everything inside a namespace (I chose :daily) but you can pick anything to help make it more descriptive. Then, the task line should be completely copied. The ":environment do" part just tells rails to utilize models and the env. while running rake.

require 'capybara/poltergeist'

namespace :daily do

  desc "Scrapes hourly for much needed dataaaaa"
  task :scrape => :environment do
    Capybara.register_driver(:poltergeist) { |app| Capybara::Poltergeist::Driver.new(app, {js_errors: true, inspector: true, debug: false})}  #, extensions: ["vendor/poltergeist/phantom_config.js"]}) }
    Capybara.default_max_wait_time = 12
    Capybara.default_driver = :poltergeist  # configure Capybara to use poltergeist as the driver
    Capybara.javascript_driver = :poltergeist
    Capybara.current_driver = :poltergeist
    @session = Capybara.current_session

    EnvisionAccount.find_each do |envision_account|
     # account = Scraper::HourlyScraper.new
      daily_sign_in(envision_account)
    end
  end    
end

Once your code is setup like this you can run your scraper from the command line by doing: bundle exec rake daily:scrape If everything works you just need to go onto Heroku, add the scheduler Addon and then tell it to run that rake command on whatever interval you need. Also note, Heroku needs a buildpack if you're using capybara and poltergeist for scraping so run this:

heroku buildpacks:add --index 1 https://github.com/stomita/heroku-buildpack-phantomjs

Enjoy