How to upload file from Google App Engine to Google Cloud Storage using Python Flask?

Question

I want to upload file from Google App Engine to Google Cloud Storage I'm using Pyhton 3.8 and Flask.

app.yaml:

runtime: python38

requirements.txt

Flask==2.0.0;
google-cloud
google-cloud-storage

I tried to upload file using Flask (https://www.tutorialspoint.com/flask/flask_file_uploading.htm) to /tmp (in App Engine temporary storage (https://cloud.google.com/appengine/docs/standard/python3/using-temp-files))

Then I tried to upload from temporary storage to Google Cloud Storage using this code samples: (https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-code-sample)

but that gives me this error:

OSError: [Errno 30] Read-only file system: 'file.xlsx'

I tried from this sample too:(https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/api-client/main.py) and give me the same error.

Then I tried with gsutil (https://cloud.google.com/storage/docs/uploading-objects#gsutil) using subprocess.check_output (Return value of x = os.system(..)) still give me the same error.

Can anyone help me? What should I do ?

#################################################

Solved

#################################################

Thank you Edo Akse , my mistake was to use "/tmp" instead of "/tmp/", thanks a lot for helping me.

the following is the complete code which has worked.

from flask import Flask, request
from werkzeug.utils import secure_filename
from google.cloud import storage
import os

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = "/tmp/"

@app.route('/')
def index():
    page = '''
    <form action="/upload" method="post" enctype="multipart/form-data">
        <input name="file" class="custom-file-input" type="file">
        <button type="submit" class="btn btn-success">Submit</button>
    </form>
    '''
    return page

@app.route('/upload',methods=["POST"])
def upload():
    try:
        if request.method == 'POST':
            f = request.files['file']
            f.save(os.path.join(app.config['UPLOAD_FOLDER'],secure_filename(f.filename)))
            
            #Start Do Main Process Here
            TMP_FILE_NAME = os.path.join(app.config['UPLOAD_FOLDER'],secure_filename(f.filename))
            BUCKET_NAME = 'fsample123'
            result = upload_blob(BUCKET_NAME, TMP_FILE_NAME, f"tmp/{secure_filename(f.filename)}")
            #End Do Main Process Here
            return result
            
    except Exception as e:
        return str(e)

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # SOURCE:
    # https://cloud.google.com/storage/docs/uploading-objects#uploading-an-object

    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)
    blob.upload_from_filename(source_file_name)
    return "File {} uploaded to {}.".format(source_file_name, destination_blob_name)

if __name__ == '__main__':
   app.run()

I think your issue might not be with the GCS library, but with the location where the temp file is stored. You don't specify GAE Flex or Standard, but both deal with temporary file storage. See this doc for Standard for example — Edo Akse
Just noticed you deal with GAE Standard, so the link above applies... — Edo Akse
yes i'm using GAE Standard, so what should i do to upload from GAE to GCS ? if the problem is the temp file storage where should i place my file for temporary to upload to GCS ? — Fian Julio

Edo Akse Edo Akse · Accepted Answer · 2021-09-09T10:16:07

So the issue is not with uploading to GCS but with the temporary file you create.

The only directory that has write access is /tmp and be aware that it is not shared between GAE instances and it disappears when the instance is restarted...

from google.cloud import storage


def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # SOURCE:
    # https://cloud.google.com/storage/docs/uploading-objects#uploading-an-object

    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)
    blob.upload_from_filename(source_file_name)
    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )


TMP_PATH = "/tmp/"
TMP_FILE_NAME = f"{TMP_PATH}file.xlsx"
BUCKET_NAME = '<your-bucket-name>'


with open(TMP_FILE_NAME, "w") as outfile:
    outfile.write("write file action goes here")
upload_blob(BUCKET_NAME, TMP_FILE_NAME, "Target file path goes here...")

How to upload file from Google App Engine to Google Cloud Storage using Python Flask?

1 Answers