gsutil cp succeeds but upload_from_filename() method from storage client fails

Question

I want to upload a file to GCS using the google storage client in python but its failing with a permissions issue, however gsutil cp succeeds. I can't understand why.

Here is what I have run using gsutil:

BUCKET=abc
$ gcloud iam service-accounts keys create --iam-account $ACCOUNT key_file.json
created key [5006838b5984f1d3b4de6523239e9bbd2c7f7047] of type [json] as [key_file.json] for [[email protected]]
$ gcloud auth activate-service-account --key-file key_file.json
Activated service account credentials for: [[email protected]]
$ touch test.txt
$ gsutil cp test.txt gs://${BUCKET}/test.txt     
Copying file://test.txt [Content-Type=text/plain]...
/ [1 files][    0.0 B/    0.0 B]                                                
Operation completed over 1 objects.

Here is what I have attempted from python:

from google.cloud import storage
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key_file.json' #same file as downloaded above
storage_client = storage.Client()
bucket_name = "abc" #the bucket name
source_file_name = "test.txt"
destination_blob_name = f"{source_file_name}"
with open(source_file_name, 'w') as f:
        f.write("lorem ipsum")
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(source_file_name)

That upload_from_filename() function fails with error:

google.api_core.exceptions.Forbidden: 403 POST https://storage.googleapis.com/upload/storage/v1/b/abc/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)

I'm baffled. I would have thought that if this worked from gsutil it would work using the python storage client. Any suggestions would be most welcome.

Will this python script run outside Google Cloud at the end (on prem, on other cloud)? — guillaume blaquiere
Yes, but at the end? It will also run on your laptop? Where do you plan to deploy this script? — guillaume blaquiere
ah I see. This script is intended to run as a test in a CI pipeline. The test will run to verify that $ACCOUNT has appropriate permissions to carry out the things it needs to be able to carry out. — jamiet
Hi, thanks for your responses. I've solved the problem. Please see the answer I posted. — jamiet

jamiet jamiet · Accepted Answer · 2020-11-07T18:26:52

I've gotten to the bottom of this problem. It was twofold.

I discovered that the code would succeed if the storage object destination_blob_name did not already exist. If the object did exist then the error explained above would occur.
Permission for $ACCOUNT to access the bucket was provided by a custom role that was applied to the bucket. When I added permission storage.objects.get & storage.objects.delete to that custom role then my code succeeded, even if the object already existed.

I've discovered that storage.objects.get & storage.objects.delete are described as

Read object data and metadata, excluding ACLs

Delete objects

https://cloud.google.com/storage/docs/access-control/iam-permissions#object_permissions

The learning I've taken away from this is that under certain circumstances (e.g. when the destination object already exists) then gsutil cp requires different permissions to google.cloud.storage.blob.upload_from_filename(). If anyone out there can elucidate the difference between those two operations then I'd love to understand it better.

gsutil cp succeeds but upload_from_filename() method from storage client fails

2 Answers