Make wget retry original URL after 3XX Redirect

1
votes

I have a service that redirects users to temporary pre-signed AWS downloads. These are large files, often 5-10gb. To prevent download sharing, we have a relatively short (30 seconds) valid lifespan.

Everything is working except that on slow internet connections, they tend to fail or get interrupted. wget has a feature that automatically retries the download. However, instead of retrying the original URL (eg: http://service.com/download/file.zip), wget retries the redirected pre-signed URL (eg: http://service.s3.amazonaws.com/file.zip?AWSAccessKeyId=XXXX&Signature=XXXX&Expires=1468000000)

Since these are large files, and the pre-signed lifespan is so short, that temporary url is no longer valid and the user gets a 403 Forbidden result.

Originally, when we noticed the problem, we were using 302 Found temporary redirects. A little research seemed to indicate we SHOULD have been using 307 Temporary Redirect. However, that didn't resolve the problem with wget. For grins and giggles, we tried 303 See Other, but that didn't work either.

Does anyone have any idea how get wget to retry the original URL instead of the redirected URL?

below is a wget example log:

--2016-07-06 10:29:51-- https://service.com/download/file.zip

Connecting to service.com (service.com)|10.0.0.1|:443... connected.

HTTP request sent, awaiting response... 302 Found

Location: https://service.s3.amazonaws.com/file.zip?AWSAccessKeyId=XXXX&Signature=XXXX&Expires=1468000000 [following]

--2016-07-06 10:29:52-- https://service.s3.amazonaws.com/file.zip?AWSAccessKeyId=XXXX&Signature=XXXX&Expires=1468000000

Resolving service.s3.amazonaws.com (service.s3.amazonaws.com)... 54.231.12.129

Connecting to service.s3.amazonaws.com (service.s3.amazonaws.com)|54.231.12.129|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2070666907 (1.9G) [application/zip]

Saving to: ‘file.zip’

file.zip 53%[=========> ] 1.03G --.-KB/s in 18m 7s

2016-07-06 10:47:59 (995 KB/s) - Read error at byte 1107205784/2070666907 (The specified session has been invalidated for some reason.). Retrying.

--2016-07-06 10:48:00-- (try: 2) https://service.s3.amazonaws.com/file.zip?AWSAccessKeyId=XXXX&Signature=XXXX&Expires=1468000000

Connecting to service.s3.amazonaws.com (service.s3.amazonaws.com)|54.231.12.129|:443... connected.

HTTP request sent, awaiting response... 403 Forbidden

2016-07-06 10:48:01 ERROR 403: Forbidden.

2
httpamazon-s3wget

2 Answers

0
votes

I had a similar issue, and a similar answer as @panzerito, but broke it up into a script i called loopdone

#!/bin/bash
until `$1`; do sleep 1; echo restarting; done

then I can just do loopdone "wget -c http://my.url/" (incl quotes) to force it to run again and again (and resume, unless server does not support it) until exit code is 0. (meaning no error)

-1
votes

Bash-code: initial_error_EXIT_STATUS; until [ "$?" -eq "0" ]; do wget https://example.com/download/file.zip -c; done