Heroku Puma Sidekiq Running 167% crititcal r14 memory quota exceeded on 3 web dynos with one visitor

Question

I am running Puma 3.4, Rails 4.2.6, Sidekiq 4.2.2, Redis Mini, on 3 dynos (standard 1x) on Heroku. I am doing simple Beta testing and I cannot seem to optimize my puma-dyno situation. I get High Response Time Alerts and Critical Memory Quota alerts when one user is on the site (I am using New Relic to monitor).

I have added Puma Worker Killer to try to diagnose this, but no help.

I have everything set using Environment variables (max_threads, concurrency, etc) and I've scoured the internet for how to config.

I am setting options in database.yml, sidekiq.rb, puma.rb, puma_worker_killer.rb and sidekiq.yml, so I might be setting things in too many places.

Since I'm running 3 Standard 1x web dynos for one user I know that something is wrong.

config/puma.rb

before_fork do
  require 'puma_worker_killer'
  ActiveRecord::Base.connection_pool.disconnect!

  PumaWorkerKiller.config do |config|
    config.ram           =  ENV['PUMA_WORKER_KILLER_RAM'] || 1024 # mb
    config.frequency     = 5    # seconds
    config.percent_usage = 0.98
    config.rolling_restart_frequency = 12 * 3600 # 12 hours in seconds
  end

  PumaWorkerKiller.start
end

workers Integer(ENV['WEB_CONCURRENCY'] || 5)

min_threads_count = Integer(ENV['MIN_THREADS'] || 1)

threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)

threads min_threads_count, threads_count

preload_app!

rackup      DefaultRackup
port        ENV['PORT']     || 5000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do


  ActiveSupport.on_load(:active_record) do
    ActiveRecord::Base.establish_connection
  end
end

config/initializers/sidekiq.rb

require 'sidekiq'

redis_url = ENV['REDISTOGO_URL']

redis_config = {
  url: redis_url,
  namespace: 'oct',
}

Sidekiq.configure_server do |config|
  config.redis = { url:  ENV["REDISTOGO_URL"], namespace: 'oct', size:  ENV["SIDEKIQ_SERVER_CONNECTIONS"].to_i || 6}

  config.error_handlers << Proc.new do |exception, context_hash|
    SidekiqErrorService.new(exception, context_hash).notify
  end
end

Sidekiq.configure_client do |config|
  config.redis = { url:  ENV["REDISTOGO_URL"], namespace: 'oct', size:  ENV["REDIS_CLIENT_CONNECTION_SIZE"].to_i || 2}
end

config/sidekiq.yml

:verbose: false

:concurrency: <%= ENV["WEB_CONCURRENCY"] %>

production:
  :timeout: <%= ENV["SIDEKIQ_TIMEOUT"] %> 
development:
  :timeout: 30

:queues:
  - [highest, 2]
  - medium
  - lowest
  - mailers

config/initializers/puma_worker_killer.rb

PumaWorkerKiller.enable_rolling_restart

config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV["DB_POOL"] || ENV['RAILS_MAX_THREADS'] || 5 %>

development:
  <<: *default
  database: myapp_development
  username: myapp
  password: myapp
  host: localhost
  port: 5432
test:
  <<: *default
  database: myapp_test
  username: myapp
  password: myapp
  host: localhost
  port: 5432

my current environment settings

Memory 125-167% (on web dynos, not worker dynos)

WEB_CONCURRENCY:  4
DB_POOL: 15
SIDEKIQ_SERVER_CONNECTIONS: 25
MIN_THREADS: 1
RAILS_MAX_THREADS: 5
REDIS_CLIENT_CONNECTION_SIZE: 1
SIDEKIQ_TIMEOUT: 30
DATABASE_REAP_FREQ: 5
PUMA_WORKER_KILLER_RAM: 1535

I also tried to use this helper http://manuelvanrijn.nl/sidekiq-heroku-redis-calc/

but it made it worse, 300% + memory usage

WEB_CONCURRENCY:  45
DB_POOL: 15
SIDEKIQ_SERVER_CONNECTIONS: 47
MIN_THREADS: 1
RAILS_MAX_THREADS: 5
REDIS_CLIENT_CONNECTION_SIZE: 1
SIDEKIQ_TIMEOUT: 30
DATABASE_REAP_FREQ: 5
PUMA_WORKER_KILLER_RAM: 1535

RangerRanger RangerRanger · Accepted Answer · 2016-11-04T18:46:27

For a Rails 4, you have too many puma workers even at 4 (45 is impossibly high for that tier). Standard-1x dynos have 512MB. Trying dropping WEB_CONCURRENCY down to 2 and I suspect that you'll see a large enough reduction in memory consumption to prevent R14s.

If you're still having trouble after switching WEB_CONCURRENCY to 2 you will need to do some memory optimizations or drop it down to 1.

Heroku Puma Sidekiq Running 167% crititcal r14 memory quota exceeded on 3 web dynos with one visitor

1 Answers