Google Cloud Storage, Compute Engine and Insufficient Permission error

Question

I know this is a widely discussed issue since 2016, but I haven't been able to fix the problem. For a Google Cloud project my personal account and a service account (the one I get with the command gcloud auth list) are both Owner.

A python3 code which use the google-cloud package (storage module) works fine on my laptop using my personal account, i.e. the file uploading works correctly.

Instead, on a compute engine, I get the following error:

  File "/home/user/git/project/GoogleUtils/StorageUtils.py", line 109, in upload_blob
    blob.upload_from_filename(source_file_name)
  File "/home/analysis/.local/lib/python3.5/site-packages/google/cloud/storage/blob.py", line 992, in upload_from_filename
    size=total_bytes)
  File "/home/analysis/.local/lib/python3.5/site-packages/google/cloud/storage/blob.py", line 949, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/home/analysis/.local/lib/python3.5/site-packages/google/cloud/storage/blob.py", line 1735, in _raise_from_invalid_response
    raise exceptions.from_http_response(error.response)
google.api_core.exceptions.Forbidden: 403 POST https://www.googleapis.com/upload/storage/v1/b/bucket_name/o?uploadType=multipart: Insufficient Permission

This happens with both my personal account and the service account. I suppose that it's not a permission problem... So where I should look for? The weird part is that the file downloading works perfectly.

Answer 1

The only solution I foun, is to modify the VM settings by switching (on the settings screen's bottom) from 'default access' to 'full access'.

Answer 2

It has been already answered, but I think it worth to clarify a bit.

Yes, you are right it is not a permission error of the service account and it is the expected behaviour due to the default settings, but let me explain a bit:

• When you create a virtual machine with Google Cloud Engine using the default service account you have to double check which are the allowed API for that VM. (you can check them and modify at the moment of the creation of an instance and you can visualise them from the Google Cloud Console checking the details of any instance). This behaviour, that might seem tricky, is intended in order to restrict the power of a service account that is possibly spread on many different instances that if compromised (it is enough to log in into the instance) can cause many issue if the users give him full powers, i.e. project owner.

• On the other hand when you do not use the Default service account you have to Use IAM roles with service accounts to control VM access to modify the access scopes. So in your case a service account having full access would be enough to use all the API available.

You can verify these two different behaviours creating a new service account and at the moment of the creation of an instance use the default account first and then the new one, you will notice that two different tabs will appear.