Signed URLs on GAE with Python for GCS PUT request

Question

I'm trying to create a signed URL to be used for uploading files directly to Google Cloud Storage (GCS). I had this working using POST using this Github example, which makes use of a policy. Per best practice, I'm refactoring to use PUT and getting a SignatureDoesNotMatch error:

<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>PUT


123456789
/mybucket/mycat.jpg</StringToSign></Error>

Per the docs on creating a signed URL with a program and the GCP example Python code, I am doing this process:

1. building my Signature string
2. signing it
3. base64 encoding it
4. url encoding the result (the python example doesn't do this though...

Since this is running on a Google App Engine (GAE) App, I shouldn't need to get a JSON key file for my service account user, but rather use App Identity Services to sign it. Here's my code within a Flask project:

google_access_id = app_identity.get_service_account_name()
expires = arrow.utcnow().replace(minutes=+10).replace(microseconds=0).timestamp
resource = '/mybucket/mycat.jpg'

args = self.get_parser.parse_args()
signature_string = 'PUT\n'
# take MD5 of file being uploaded and its content type, if provided
content_md5 = args.get('md5') or ''
content_type = args.get('contenttype') or ''
signature_string = ('PUT\n'
                    '{md5}\n'
                    '{content_type}\n'
                    '{expires}\n'
                    '{resource}\n').format(
                    md5=content_md5,
                    content_type=content_type,
                    expires=expires,
                    resource=resource)
log.debug('signature string:\n{}'.format(signature_string))
_, signature_bytes = app_identity.sign_blob(signature_string)
signature = base64.b64encode(signature_bytes)
# URL encode signature
signature = urllib.quote(signature)
media_url = 'https://storage.googleapis.com{}'.format(resource)
return dict(GoogleAccessId=google_access_id,
            Expires=expires,
            Signature=signature,
            bucket='mybucket',
            media_url='{}?GoogleAccessId={}&Expires={}&Signature={}'.format(media_url, google_access_id, expires, signature))

The log.debug statement prints a signature file which perfectly matches the signature in the GCS XML error response above. If they match, then why can't I upload?

Using gsutil, I can create a signed URL using the same GAE service account, and it works fine in Postman. I see gsutil URL-encodes the signature, but when creating my own signed URL, it doesn't seem to matter either way: GCS gets my PUT request and complains that the signature doesn't match, even though the signature it shows me matches my logged debug message. I've also tried with and without a trailing \n in the original signature string.

EDIT: The POST example I followed Base64 encodes the Policy before it sings, and again after it signs it. I tried this approach with the PUT signature creation and it made no difference

Answer 1

The answer was very close to other answers found on SO and elsewhere noting that a Content-Type header needs to be used. This is partly true, but overshadowed by my main problem: I was relying on the Default GAE Service Account which has permissions of "Editor" which I assumed could read and write to GCS. I made a keyfile from that account and used it with gsutil which then gave me this clue:

[email protected] does not have permissions on gs://mybucket/cat.jpg, using this link will likely result in a 403 error until at least READ permissions are granted

It was right that I'd get an error from GCS when trying to PUT a file there, but it wasn't a 404 error, rather it was the SignatureDoesNotMatch error shown in the question.

The solution was 2 part:

1. Give the Default GAE service account more permission from the "Storage" set of permissions:
   1. Storage Object Creator
   2. Storage Object Viewer
2. Be sure to use Content-Type Header when PUTting the file to GCS because even I don't specify it in the Signature string to be signed, it will default to text/plain and require that in the Header.

One final observation: Even after I added the correct permissions to my account, I still got the warning with gsutil until I created a new JSON key file from the IAM console. Furthermore, signed URLs made with gsutil and the old key file failed to work (SignatureDoesNotMatch error) even if the permissions were set on the IAM console. The Python code running on GAE worked fine without any updates - it only needed the permissions set in IAM and then the Content-Headers to match.