Shell script wget download from S3 - Forbidden error

Question

I am trying to download a file from Amazon's S3 using a shell script and the command wget. The file in cuestion has public permissions, and I am able to download it using a standard browsers. So far this I what I have in the script:

wget --no-check-certificate -P /tmp/soDownloads  https://s3-eu-west-1.amazonaws.com/myBucket/myFolder/myFile.so
cp /tmp/soDownloads/myFile.so /home/hadoop/lib/native

The problem is a bit odd for me. While I am able to download the file directly from the terminal (just typing the wget command), an error pops up when I try to execute the shell script that contains the very same command line (Script ran with >sh myScript.sh).

--2014-06-26 07:33:57--  https://s3-eu-west-1.amazonaws.com/myBucket/myFolder/myFile.so%0D
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... XX.XXX.XX.XX
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|XX.XXX.XX.XX|:443... connected.
WARNING: cannot verify s3-eu-west-1.amazonaws.com's certificate, issued by     ‘/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3’:
Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 403 Forbidden
2014-06-26 07:33:57 ERROR 403: Forbidden.

Now, I am aware this can just be a begginer error from my side, but I am not able to detect any mispelling or error of any type. I would appreciate any help you can provide me to solve this issue.

As a note, I would like to notice that I am running the script in a EC2 instance provided by Amazon's Elastic MapReduce framework, if it has something to do with the issue.

Michael - sqlbot Michael - sqlbot · Accepted Answer · 2014-06-26T20:03:19

I suspect that the editor you used to write that script has left you a little "gift."

The command line isn't the same. Look closely:

--2014-06-26 07:33:57-- ... myFolder/myFile.so%0D
                                              ^^^ what's this about?

That's urlencoding for ASCII CR, decimal 13 hex 0x0D. You have an embedded carriage return character in the script that shouldn't be there, and wget is seeing it as the last character in the URL, and sending it to S3.

Using the less utility to view the file, or an editor like vi, this stray character might show up as ^M... or, if they're all over the file, with you open it with vi, you should see this at the bottom of the screen:

"foo" [dos] 1L, 5C    
      ^^^^^

If you see that, then inside vi...

:set ff=unix[enter] 
:x[enter]

...will convert the line endings, and save the file in what should be a usable format, if this is really the problem you're having.

If you're editing files on windows, you'll want to use an editor that understands how to save files with newlines, not carriage returns.

Shell script wget download from S3 - Forbidden error

1 Answers