Uploading large video file to Google App Engine

Question

I am trying to upload a large video into Google Cloud Storage via Google App Engine.

I followed the principles in this post which refers to uploading a large spreadsheet. Can't upload large files to Python + Flask in GCP App Engine

I have set up chunking using dropzone.js

I have set up an upload in main.py which I had hoped would upload the file chunks into the app's tmp directory and have put in logic which moves the completed file into Google Cloud Storage once all parts are in place.

I get the following error message:

TypeError: write() argument must be str, not bytes

Here is my back end code


    from flask import Flask, render_template, request, redirect, url_for
    from google.cloud import storage
    from flask_dropzone import Dropzone
    from werkzeug.utils import secure_filename
    import os
    import base64


    app = Flask(__name__, template_folder='./templates', static_folder="./static")
    dropzone = Dropzone(app)
    app.config['UPLOAD_PATH'] = '/tmp'

    @app.route('/', methods=['GET', 'POST'])

    def index():
        return render_template('index.html')


    @app.route('/upload', methods=['POST', 'GET'])
    def upload():

        if request.method == 'POST':

        upload_file = request.files.get('file')

        tmp_file_path = '/tmp/' + upload_file.filename

        with open(tmp_file_path, 'a') as f:
            f.write(uploaded_file.read())

        chunk_index = int(flask.request.form.get('dzchunkindex')) if 
(flask.request.form.get('dzchunkindex') is not None)  else 0
    
        chunk_count = int(flask.request.form.get('dztotalchunkcount')) if (flask.request.form.get('dztotalchunkcount') is not None)  else 1


        if (chunk_index == (chunk_count - 1)):
            print('Saving file to storage')
            print( chunk_count )

            storage_client = storage.Client()

            storage_bucket = storage_client.get_bucket('percy-277618.appspot.com')

            blob = storage_bucket.blob(upload_file.filename)

            blob.upload_from_filename(tmp_file_path, client=storage_client)
            print('Saved to Storage')

            print('Deleting temp file')
            os.remove(tmp_file_path)

                  

if __name__ == '__main__':

    app.run(host='127.0.0.1', port=8080, debug=True)

This is my front end code.






Dropzone.options.uploadwidget = {

       paramName: 'file',
       forceChunking: true,
       timeout: 300000,
       chunking: true,
       url: '/upload',
       chunkSize: 10485760,
       maxFilesize: 1025,

};

I found some StackOverflow links 1, 2 that explain the cause of this error and might help you. The suggestions that are included is to open files in binary mode (include 'b' in mode argument) or convert bytes to str. I hope this helps and your error gets eliminated. — Artemis Georgakopoulou

Dustin Ingram Dustin Ingram · Accepted Answer · 2020-08-05T15:28:11

Using uploaded_file.read() produces bytes, not a string. You should open the file in binary mode:

with open(tmp_file_path, 'ab') as f:

Uploading large video file to Google App Engine

1 Answers