Why does upload_from_file Google Cloud Storage Function Throws timeout error?

Question

I created a function that works for me, It uploads a file to Google Cloud Storage.

The problem is when my friend tries to upload the same file to the same bucket using the same code from his local machine, he gets timeout error. His internet is very good and he should be able to upload the file with no problems within his connections.

Any idea why is this happening?

def upload_to_cloud(file_path):
    """
        saves a file in the google storage. As Google requires audio files greater than 60 seconds to be saved on cloud before processing
        It always saves in 'audio-files-bucket' (folder)
        Input:
            Path of file to be saved
        Output:
            URI of the saved file
    """
    print("Uploading to cloud...")
    client = storage.Client().from_service_account_json(KEY_PATH)
    bucket = client.get_bucket('audio-files-bucket')
    file_name = str(file_path).split('\\')[-1]
    print(file_name)
    blob = bucket.blob(file_name)
    f = open(file_path, 'rb')
    blob.upload_from_file(f)
    f.close()
    print("uploaded at: ", "gs://audio-files-bucket/{}".format(file_name))
    return "gs://audio-files-bucket/{}".format(file_name)

It throughs the timeout exception in upload_from_file(f) line.

We tried to use upload_from_filename function, but the same error still occurs.

Does your friend have set up the same environment as you? i.e. he have the json key for the same service account, he is trying to upload the same file as you, he have a folder named 'rb' in the same path as you? — Chris32
@Chris32 He has the same key yes. Trying the same file, it is 12.5 MB. The rb stands for read binary not a folder. The file path is passed as a parameter to the function. — Raamyy
Thanks for your reply Raamy. Would be interesting to know more differences between both environments i.e. different SO? if this is the case (i.e. you using windows and he using linux to run the command) Can your frind please include the print of the file_name variable please? It looks like he's getting a lot of file at once since the timeout for uploads it's 60 seconds and for 12.5 mb it should be possible with an internet connection of arround 2Mbps — Chris32
@Chris32 Sorry for the late reply we were still experimenting. The problem is solved by reducing the default chunk size of the storage blob. And it worked with him. Thanks for the help! — Raamyy

Raamyy Raamyy · Accepted Answer · 2020-04-05T19:25:38

The problem is solved by reducing the chunk size of the blob. The code changed to be:

def upload_to_cloud(file_path):
    """
        saves a file in the google storage. As Google requires audio files greater than 60 seconds to be saved on cloud before processing
        It always saves in 'audio-files-bucket' (folder)
        Input:
            Path of file to be saved
        Output:
            URI of the saved file
    """
    print("Uploading to cloud...")
    client = storage.Client().from_service_account_json(KEY_PATH)
    bucket = client.get_bucket('audio-files-bucket')
    file_name = str(file_path).split('\\')[-1]
    print(file_name)
    blob = bucket.blob(file_name)

    ## For slow upload speed
    storage.blob._DEFAULT_CHUNKSIZE = 2097152 # 1024 * 1024 B * 2 = 2 MB
    storage.blob._MAX_MULTIPART_SIZE = 2097152 # 2 MB

    with open(file_path, 'rb') as f:
        blob.upload_from_file(f)
    print("uploaded at: ", "gs://audio-files-bucket/{}".format(file_name))
    return "gs://audio-files-bucket/{}".format(file_name)

Why does upload_from_file Google Cloud Storage Function Throws timeout error?

1 Answers