How to handle incoming emails with multiple large attachments from a Rails application?

Question

I need to setup a Ruby / Rails application which can accept incoming emails, the body of the emails will be ignored, each email will have one or more large attachments (graphics) up to 20MB in size, the app will receive approx 10-20 of these emails an hour.

I need to use a custom email address / domain.

Quick processing of the emails is not a big priority, however each email should get handled in at most 5 minutes after it arrives.

I was thinking of having all emails automatically forwarded from my info@mydomain.com mail account to a GMail inbox, then have a Rails lib script use MMS2R & TMail access GMail, download any new mails & add them to a message queue (delayed job), this script would be invoked by a cron job once a minute.

This solution just does not "smell right" & I would be concerned about its ability to scale; I don't want to load up the whole Rails stack each time the cron / script is run.

Does anyone have experience with incoming mail & attachment processing in Rails?

[...] forwarded from my info@mydomain.com mail account to a GMail inbox, then have a Rails lib script use MMS2R & TMail access GMail [...]. Why is this necessary? Now other solutions aside, why not to download the emails right from your mydomain mailbox? — Behrang
If you are concerned about the cron job loading all the rails environment each time I suggest you to take a look at link. — cicloon
@Behrang - true I could do that, more concerned though with how to have my app access whichever mail account I use & efficiently process the attachements, would prefer not to spin up a rails instance once per minute or each time the script is run. Ideally I would like the have the incoming emails / attachements sent as a POST request to a controller....but due to the large attachment sizes, this might to be slow if many requests came in at once. — Jason
@Jason: Do you need to have access to the Rails environment for processing the attachments? If not, you can use a background job handler like Starling, Beanstalkd, Resque, etc... — Behrang

rubish rubish · Accepted Answer · 2011-08-30T06:38:47

If your main concern is about loading rails environment every minute to run the script, I would suggest mailgun along with some cron alternatives.

I use a combination of bluepill + clockwork + delayed_job to handle this. I have custom jobs, enqueing which is as simple as Delayed::Job.enqueue MyJob.new. The logic is captured in job's perform method, which helps minimize dependencies if there are any.

clockwork is used to enqueue delayed jobs only loading minimal dependencies, keeping the memory footprint minimal.

bluepill is used to monitor the delayed workers and clockwork process and start them back up if they fail or restart if they are getting out of control.

General memory usage: 20MB(bluepill) + 25MB(clockwork) + 80-90MB(delayed worker)

Example of clockwork.rb:

ENV['RAILS_ENV'] ||= "development"
ENV['RACK_ENV'] = ENV['RAILS_ENV']

require 'clockwork'
require 'mongoid'
require 'delayed_job'
require 'delayed_job_mongoid'

include Clockwork

base_directory = File.absolute_path(File.dirname(__FILE__))
require File.join(base_directory, "../jobs/fetch_comments_job.rb")
require File.join(base_directory, "../jobs/news_letter_job.rb")

Mongoid.load!(File.join(base_directory, "../config/mongoid.yml"))

every(1.minute, "pop.comments") { Delayed::Job.enqueue FetchCommentsJob.new}
every(1.day, "send.newsletter", :at => '22:30') { Delayed::Job.enqueue NewsLetterJob.new}

How to handle incoming emails with multiple large attachments from a Rails application?

2 Answers