Heroku intermittent H15 'Idle connection' error on upload action

Question

I have a heroku application running Rails 4.1.5 on Ruby 2.1.1.

The app involves a bit of media processing, and involves a javascript upload function using dropzone JS, to an action that saves the photo on Amazon S3 before later processing in a background job.

I'm having a really frustrating problem. Intermittently - and dependent on the speed of the internet connection - the Heroku Router is killing the upload action with a H15 (idle connection) status. I don't believe that the connection has really gone idle, as using the same internet connection, I can upload similar files to other sites.

This isn't related to the web dyno's timeout, as the request never hits the dyno in these cases where the router shuts it down first.

What could be happening here?

Note: I've implemented this CORS-handling technique: http://www.tsheffler.com/blog/?p=428

Logs:

[Timestamp] heroku[router]: at=error code=H15 desc="Idle connection" method=POST path="/console/breeds/57-YAMTALE/photos" host=[host] request_id=634d7680-8b66-4123-a4bc-fe78dc4e7b90 fwd="[IP]" dyno=web.1 connect=0ms service=90232ms status=503 bytes=0

Procfile:

web: bundle exec rails server thin -p \$PORT -e \$RACK_ENV

Gems that might be relevant:

gem 'thin'

gem 'carrierwave'

gem 'carrierwave_backgrounder'

gem 'rmagick', :require => 'RMagick'

gem 'aws-sdk'

gem 'rack-cors', :require => 'rack/cors'

gem 'daemons'

gem 'exception_notification'

Answer 1

You need to upload large files directly to S3 (or equivalent), not via your Heroku app. You can post the urn to your app (likely using javascript), and then process from/to S3 in the background.

This is, unfortunately, involved.

Heroku has (actually pretty good) instructions here on one particular approach to doing this: https://devcenter.heroku.com/articles/direct-to-s3-image-uploads-in-rails

Details

The Heroku router will time-out if it does not receive a first-byte back within 30 seconds. Thereafter, it will track a 55-second rolling window which is reset any time a byte is transferred up or down. https://devcenter.heroku.com/articles/request-timeout

In your case, I'm guessing your upload sometimes takes longer than 30-seconds. Although you are processing the file as a background task, I'm guessing you don't send a first-byte back until after the upload is complete. Hence the intermittent timeout.

In the same Heroku document referenced above:

Many web applications allow users to upload files. When these files are large, or the user is on a slow internet connection, the upload can take longer than 30 seconds. For this we recommend directly uploading to S3.

Chunked uploads

I believe there is an alternate way to solve this using HTTP chunked uploads where the first-byte response can be sent early on. (This would work only on Cedar 1.1 and newer which supports the 55-second rolling window)

In fact, the chunked approach should be simpler than the direct-to S3 method above. I never followed up on this for a couple of reasons:

1) Heroku does not mention it as an option.

2) It would go against Heroku's general advice:

Best practice is to get the response time of your web application to be under > 500ms, this will free up the application for more requests and deliver a high > quality user experience to your visitors.

3) It could (based on my understanding, it would) negatively impact Heroku's load balancing, which is primitive (or broken depending on who you ask).