How to use CarrierWaveDirect with embedded Mongo documents and legacy Carrierwave files

Question

I'm looking for the best ways to upload a file in a Rails app on Heroku. Setup: Rails 3, Carrierwave, Heroku, Mongoid

Problem: Sporadically, when a user uploads a file of varying sizes, the image is saved to the database, but not available in any form on Amazon S3. This results in the image displaying as a missing image placeholder on the front end.

Probable Cause: Image is uploaded but processing is too time consuming and the request times out due to Heroku's hard 30 second request time limit, or image size is too large and in attempting to upload, the request times out with H12 error.

Solution: Implement carrierwave-direct and move processing to background

Blocker: CarrierWaveDirect is not a drop in replacement for CarrierWave. In fact, it introduces some major changes in the process of uploading an image, notably the image is not uploaded at the same time of db record persistence. Also, it does not work with my existing directory structure, so moving my existing files to a new structure seems overly arduous. CarrierWaveBackgrounder does not play nice with embedded documents.

Question: How do I do this? Example code if needed, but trying to think through this moreso than just coding it.

Answer 1

If you need to handle single files, check out this Railscast which covers CarrierWaveDirect and uploading to S3 in depth. Combined with the carrierwave-mongoid gem, you should be able to get single file uploads running if you want to go that route.

Honestly though, I'd just go for multiple file upload and be done with it. In the second part he shows off a multi-file upload setup using standard Carrierwave and jQuery File Upload instead of CarrierWaveDirect. I tend to agree with that decision. I just couldn't get it working right, most likely because of all the black magic required to POST directly to S3.

Ryan's setup looks intimidating at first, but it's pretty simple, and in the source code for the episode he includes a helper class that takes care of all that form data funny business S3 requires.

The basic workflow is something like:

1) File gets uploaded directly from browser to temporary bucket or folder on S3 through jQuery plugin.

2) AJAX POST request with the temp upload URL goes to app telling it to persist in the database and start a worker process.

3) Worker process downloads temporary file, runs it through your CarrierWave setup and stores the finished versions in the right place.

I think he uses sidekiq for the background jobs, but I have something similar set up with IronWorker and it works beautifully. Once you have the jQuery stuff and CarrierWave set up, you can use whatever you want for the jobs really.

Also, in the comments there I explained how I take care of deleting the raw upload files from the temp bucket as part of the job to keep things tidy. And for all the dirty details on how POSTing straight to S3 works see here. Still a good idea to understand that even if you have the helper class.