3
votes

We recently moved a Nexus instance to AWS and are having trouble with large file uploads being closed. We suspect this may be due to the ELB timeout due to this gem:

"If an HTTP request doesn't complete within the idle timeout period, the load balancer closes the connection, even if data is still being transferred. "

Source: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/config-idle-timeout.html

How can a connection be idle if it is transferring data?? Why would it be done this way?

Some of the files are many gigabytes - it may take several minutes to upload, let's say 30 minutes if busy. How are we supposed to support this, is setting the timeout to 1800s really the recommended fix? (max 3,600)

Thanks, Joel

2
Keep-alive is enabled at server and client, but that only has effect with sequential requests. If there are other possible causes and solutions I'd be happy to hear about it! :) - jbyrnes
Are you uploading to s3? You allude to that in a comment below. - BestPractices
No, the comment is meant to say that we are not allowed to use S3 as had been suggested, for reasons too complicated to go into. - jbyrnes

2 Answers

2
votes

That does indeed appear to be an amusing gem. The explanation likely lies in the underlying implementation actually having two timers, one for the client and one for the server. I'm speculating, but if not for something along those lines, certain denial of service attacks could be easier to implement against ELB and the machines behind it.

The answer will lie in the ELB access logs. If you see times suspiciously close to 60 seconds, you have a likely culprit.

Increasing the timer may be one option.

Typically, though, it seems like web apps need to be in and out, over and done, moving on to whatever is next, within milliseconds. Tying up a process or a thread with something long-running like an upload means hundreds or thousands of other requests you might have been able to handle, if not for that resource hog of an upload. File uploads of really large files might be better handled by delegating them to a separate environment or service (such as S3, which can accept a POST upload, then redirect the browser back to your "success" page when the upload is done). Other tactics might include smart client logic to send the upload in parts, maybe in parallel, with the ability to restart/retry and do other clever things like progress bars.

I have probably 20-30 ELB deployments at ${dayjob}, and this has never come up, but then I don't have any systems that deal in "large" files on the user-facing side. A "large" upload for these systems that I am thinking of might be 16MB, so definitely a different scale.

1
votes

Yes, you are hitting the ELB timeout default of 60 seconds. The workaround is to increase the ELB timeout to something that is high enough where your uploads won't fail (current max: 1 hour). The connection appears ideal because the request is not completing within the timeout period.