403 Error uploading data to Google Storage with Python SDK after calling fork()

Question

I am trying to upload content to a Google bucket using Python SDK in a forked process on a Linux host (Ubuntu 18.04) The operation fails with status code 403:

google.api_core.exceptions.Forbidden: 403 POST https://www.googleapis.com/upload/storage/v1/b/temp-compare/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', )

Example code:

import random
import string
import json
import os
import requests
from datetime import datetime, timedelta
from google.cloud import storage

def upload_to_bucket(plainText):
    gcp_client = storage.Client.from_service_account_json("/path/to/google-bucket-credentials.json")
    bucket = gcp_client.get_bucket('bucket_name')
    bucket_file_name = random_file_name('json', 10)
    blob = bucket.blob(bucket_file_name)
    blob.upload_from_string(plainText)
    url = blob.generate_signed_url(expiration=datetime.utcnow() + timedelta(hours=48), method="GET")
    return url

def random_file_name(ext, length):
    return "%s.%s" % (''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(length)), ext)


if __name__ == "__main__":
    os.fork()
    url = upload_to_bucket("just for test")
    res = requests.get(url)
    print(res.content.decode('utf8'))

I have tried using the solutions offered in google-api-python-client broken because of OAuth2? with no lock.

How can this be solved?

Erez Ben Harush Erez Ben Harush · Accepted Answer · 2019-04-30T13:49:42

Not surprisingly it turns out i'm the source of the problem. After forking the process the random filename created by random_file_name is the same (probably has something to do with the way a seed is generated) Since the account does not have permission to override the file it gets a 403 response.

If I change random_file_name to be:

def random_file_name(ext, length):
    import uuid
    return  str(uuid.uuid4())[0:length]+"."+ext

All is working as expected.

I would probably be still looking for a solution if it not for @TasosZG advise to use the cloud shell.

403 Error uploading data to Google Storage with Python SDK after calling fork()

2 Answers