Error when uploading from a Cloud Storage bucket to Google Colab using 'gsutil'

Question

I've trained a model in a Compute Engine VM instance in GCP and copied the weights into a Cloud Storage bucket using the gsutil cp -r command.

I then made the bucket public and tried to copy those weights into a Google Colab notebook using the command !gsutil cp -r gs://{bucket/folder} ./

However, I get the following error:

ResumableDownloadException: Transfer failed after 23 retries. Final exception: Anonymous caller does not have storage.objects.get access to {folder/path}

Why am I getting this error?

Answer 1

Edit:

• The Cloud Storage bucket is missing the appropriate Cloud IAM role to make it fully publicly read from. The role roles/storage.objectViewer provides the necessary permissions to read and list objects from the bucket - assigning it to allUsers will make it public.

  Therefore, as per the documentation, this can be achieved with a single gsutil iam command:

  gsutil iam ch allUsers:objectViewer gs://[BUCKET_NAME].

  And then, in in Google Colab you should be able to read (or download) objects from Cloud Storage buckets with:

  !gsutil cp -r gs://[BUCKET_NAME]/[FOLDER_NAME] ./

  
  

• A safer approach is to instead of making the entire Cloud Storage bucket public, to authenticate with it using the following Python code in the notebook:

  from google.colab import auth auth.authenticate_user()

  Then set the project ID you're using with a gcloud command; replacing my-project accordingly:

  !gcloud config set project my-project

  And finally run the gsutil command; replacing bucket and folder:

  !gsutil cp -r gs://[BUCKET_NAME]/[FOLDER_NAME] ./