Deploying Rails with Resque and/or Redis in AWS Elastic Beanstalk

Question

I am trying to deploy my Rails app with Websockets-Rails in standalone mode, Resque, and Redis on AWS Elastic Beanstalk. The Ubuntu 14.04 server is running Ruby 2.2 on Puma.

Everything works fine in development mode on Puma. The errors I'm getting in production on AWS Elastic Beanstalk seem to be related to Redis.

Redis::CannotConnectError (Error connecting to Redis on my.domain:6379 (ECONNREFUSED)):
  redis (3.2.0) lib/redis/client.rb:320:in `rescue in establish_connection'
  redis (3.2.0) lib/redis/client.rb:311:in `establish_connection'
  redis (3.2.0) lib/redis/client.rb:91:in `block in connect'
  redis (3.2.0) lib/redis/client.rb:273:in `with_reconnect'
  redis (3.2.0) lib/redis/client.rb:90:in `connect'
  redis (3.2.0) lib/redis/client.rb:337:in `ensure_connected'
  redis (3.2.0) lib/redis/client.rb:204:in `block in process'
  redis (3.2.0) lib/redis/client.rb:286:in `logging'
  redis (3.2.0) lib/redis/client.rb:203:in `process'
  redis (3.2.0) lib/redis/client.rb:109:in `call'
  redis (3.2.0) lib/redis.rb:1874:in `block in hget'
  redis (3.2.0) lib/redis.rb:37:in `block in synchronize'
  /opt/rubies/ruby-2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
  redis (3.2.0) lib/redis.rb:37:in `synchronize'
  redis (3.2.0) lib/redis.rb:1873:in `hget'
  redis-objects (1.2.1) lib/redis/hash_key.rb:29:in `hget'
  /opt/rubies/ruby-2.2.3/lib/ruby/gems/2.2.0/bundler/gems/websocket-rails-cf5d59b671c5/lib/websocket_rails/synchronization.rb:184:in `block in find_user'

And sometimes I get a Redis::TimeoutError (I can't seem to reproduce this anymore).

I've added pre appdeploy scripts for Redis and Resque:

# .ebextensions/redis_server.config
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/14_redis_server.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars
      cd $EB_CONFIG_APP_ONDECK
      su -c "leader_only redis-server" $EB_CONFIG_APP_USER ||
      echo "Redis server startup failed, skipping."
      true

# .ebextensions/resque_workers.config
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/16_resque_workers.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars
      cd $EB_CONFIG_APP_ONDECK
      su -c "leader_only TERM_CHILD=1 QUEUES=* rake environment resque:work & rake environment resque:scheduler" $EB_CONFIG_APP_USER ||
      echo "Resque initialization failed, skipping."
      true

I have a suspicion that this might be due to Redis not actually deploying, but I am not sure how to check if it is or not.

What is the correct way to starting up Redis and other rake tasks like Resque upon deployment on Elastic Beanstalk?

Another possible issue is using websockets with Redis. I've read somewhere that I need to modify nginx.conf and its Upgrade header tag to permit Websockets, but I'm not sure if this is the direct cause of this problem.

EDIT:

Redis is now running on Elasticache. I'm not getting any Redis connection errors anymore, but Resque and Websockets doesn't seem to be working. I don't think it's Redis that's causing the issue, but probably isolated issues with Resque and Websockets.

I tried using a monit script to ensure Resque scheduler and workers persist:

packages:
  yum:
    monit: []

files:
  "/etc/monit.d/resque_worker":
    mode: "000644"
    owner: root
    group: root
    content: |
      check process resque_worker_QUEUE
        with pidfile /var/app/resque_worker_QUEUE.pid
        start program = "/bin/sh -l -c 'cd /var/app/current; nohup rake environment resque:scheduler PIDFILE=/var/app/resque_scheduler.pid >> log/resque_scheduler.log 2>&1' && nohup rake nohup rake environment resque:work TERM_CHILD=1 QUEUE=* VERBOSE=1 PIDFILE=/var/app/resque_worker_QUEUE.pid >> log/resque_worker_QUEUE.log 2>&1'" as uid webapp and gid webapp
        stop program = "/bin/sh -c 'cd /var/app/current && kill -9 $(cat tmp/pids/resque_scheduler.pid) && rm -f /var/app/resque_scheduler.pid && kill -9 $(cat tmp/pids/resque_worker_QUEUE.pid) && rm -f /var/app/resque_worker_QUEUE.pid; exit 0;'"
        if totalmem is greater than 300 MB for 10 cycles then restart  # eating up memory?
        group resque_workers

commands:
  remove_bak:
    command: "rm /etc/monit.d/resque_worker.bak"
    ignoreErrors: true

service:
  sysvinit:
    monit:
      ensureRunning: true
      enabled: true

Which doesn't seem to be working.

It works in development since I run the commands manually and instance aren't destroyed.

I also need to persist a standalone server for Websockets (I'm using the gem 'Websocket-Rails') on port 3001.

is there a reason you're running redis inside the beanstalk instead of using Elasticache? The problem with running redis from inside the beanstalk VMs is that redis will come up and down repeatedly. It will also have multiple, meaning your connection will go through a load balancer and end up at a random instance of redis. I don't think this will work the way you think it will. — Marc Young
I've got a Redis Elasticache up and running and it doesn't seem to be producing the error anymore. Is there any way to start up Resque workers and persist them for as long as the EB environment is up? Resque delayed/queued tasks don't seem to be going through. — Richard
simple PID management through a .pid file and you can manage the workers. I don't know the specifics for resque — Marc Young
I'm not entirely certain how to persist workers and a scheduler for Resque, on EB, especially since it auto-balances. I've edited my answer with the monit script I tried to use. — Richard

Tawan Tawan · Accepted Answer · 2016-01-07T21:19:32

I don't think it is a good idea to run a queue process within an Elastic Beanstalk web environment. I think it makes more sense to use a vanilla EC2 instance to host the queue process. However, you can also use Amazon Simple Queue Service (SQS) instead of Resque. Then you don't have to monitor and maintain the queue instance and have a very scalable solution.

If you use Resque to coordinate background jobs in a Rails >= 4.2 application, then have a look at the Active Elastic Job gem. It might solve your problem in an elegant way.

Disclaimer: I'm the author of Active Elastic Job.

Deploying Rails with Resque and/or Redis in AWS Elastic Beanstalk

1 Answers