Signing temporary s3 upload URLs w/ Haskell

Question

I'm trying to upload files from a web form directly to to Amazon S3 asynchronously. In order to do this I must authenticate the client request to upload files on the server.

By digitally signing an upload request w/ my AWS Secret key I can create a temporary authenticated URL that the client can use to upload files to a S3 bucket.

The amazon S3 docs specify that the signature must be generated by the following

Signature = URL-Encode( Base64( HMAC-SHA1( YourSecretAccessKeyID, 
                                         UTF-8-Encoding-Of( StringToSign ) ) ) );

I'm using Haskell on the server so my implementation looks like:

{-# LANGUAGE OverloadedStrings #-}
import qualified Data.ByteString.Base64.Lazy as B64
import qualified Data.Digest.Pure.SHA as SHA
import qualified Data.ByteString.Lazy.Char8 as BL8

sign :: BL8.ByteString -> BL8.ByteString
sign = B64.encode . SHA.bytestringDigest . SHA.hmacSha1 secret
  where secret = "aws-secret-key"

The format of the amazon docs requires that StringToSign look like:

StringToSign = HTTP-VERB + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Expires + "\n" +
    CanonicalizedAmzHeaders +
    CanonicalizedResource; 

Another example from Amazon:

GET\n
\n
\n
1175139620\n

/johnsmith/photos/puppy.jpg

So my string looks like:

"PUT\n\n\n1384330538\n/bucketname/objname"

I sign the string above (w/ the sign function) and craft a url that looks like:

https://s3.amazonaws.com/bucketname/objname?AWSAccessKeyId=accessskey&Signature=signature=&Expires=1384330979

This is then sent to the client via an AJAX request before an upload. I have updated the CORS policy on the bucket as well to allow for PUT requests.

The problem is that every time I try to upload something with the above signed url I get this message (in an XML doc).

The request signature we calculated does not match the signature you provided. Check your key and signing method.

So I'm not sure where I went wrong. Note: I can upload if I use the public url (https://s3.amazonaws.com/bucketname/objname) (but this shouldn't be, I only want users to upload blobs, not read nor delete, etc.)

Answer 1

As someone who's done this dance a lot, it's very difficult to build software that correctly signs an HTTP-digest authenticated request like this. In particular, if you rely only on the server response to guide you it will take a long time. For security purposes the servers are deliberately cryptic when rejecting you.

My best tip is to (a) get an alternative implementation that you know works and (b) build your Haskell interface to be pure so that it's easy to make it exactly replicate a request from that other framework and (c) make sure you can get both the exact request text and exact String-To-Sign from both the alternative framework and your own code. In particular, you'll often have to impute exact timestamps and nonces and pay close attention to percent encodings.

With these two tools just create a variety of successful requests from the alternative implementation and see if you can replicate the exact String-To-Sign and exact request text using your own framework.

Most often my own errors involved improper encoding, missing quotes, not including all of the proper parameters (or the wrong ones), or using the hmac function incorrectly.

Answer 2

here is my upload url code, i might have missed a couple of imports since i pulled it out of the deep.

{-# LANGUAGE OverloadedStrings, FlexibleContexts, TypeFamilies, DeriveDataTypeable, TemplateHaskell, QuasiQuotes #-}

import qualified Aws
import qualified Aws.Core          as Aws
import qualified Aws.S3            as S3
import qualified Data.Text         as T
import qualified Codec.Binary.Base64         as B64
import qualified Data.ByteString             as BS
import Text.Shakespeare.Text(st)
import qualified Codec.Binary.Url            as Url
import System.Posix.Time(epochTime)
import Crypto.MAC.HMAC(hmac)
import Crypto.Hash.SHA1(hash)

data Cfg = Cfg { baseCfg :: Aws.Configuration
               , s3Cfg :: S3.S3Configuration Aws.NormalQuery
               , s3Bucket :: S3.Bucket
               }

uploadUrl :: Cfg -> T.Text -> T.Text -> IO T.Text
uploadUrl cfg mime filename = do
   time <- epochTime
   let expires = show $ time + 600
       msg = E.encodeUtf8 $ [st|PUT

#{mime}
#{expires}
x-amz-acl:public-read
/#{s3Bucket cfg}/#{filename}|] --the gap is necessary
       key = Aws.secretAccessKey $ Aws.credentials $ baseCfg cfg
       accessid = T.pack $ Url.encode $ BS.unpack $ Aws.accessKeyID $ Aws.credentials $ baseCfg cfg
       signature = encode . T.pack $ B64.encode $ BS.unpack $ hmac hash 64 key msg
       encode = T.pack . Url.encode .  BS.unpack . E.encodeUtf8
   return $ [st|http://#{s3Bucket cfg}.s3.amazonaws.com/#{filename}?AWSAccessKeyId=#{accessid}&Expires=#{expires}&Signature=#{signature}|]