Google App Engine very slow response time in file upload

Question

I have a Django app running on GAE(Google App Engine) standard environment. I am uploading videos via my app to google cloud buckets (utilising dropzone on front end). The size of my videos are large so I am sending my file in chunks to GAE and then recreate and upload to GCS. It is working fine.

My issues is the request to upload the chunks are taking a long time (15s for uploading a chunk of 1MB, I removed the processing of the chunk and still taking 15s)

@login_required(login_url="/login/")
@require_http_methods(["POST"])
def upload_chunks(request):
    try:
        file = request.FILES['file']            
        # upload_utils.upload_image_file(file); //Commented it out
        return JsonResponse({"status": True})
    except Exception as e:
        return HttpResponseServerError(str(e))

Every Chunk Upload takes 15 sec and all of the time is spent in TTFB

Also when increase my chunk size to let's say 8 MB the request get cancelled as it takes more than 30 sec.

Please help!!

Answer 1

So the PHP docs offer CloudStorageTools::createUploadUrl():

User upload of files directly to Google Cloud Storage is faster and more cost-effective than writing to Google Cloud Storage from your App Engine app, because this consumes instance hours and incurs cost. Moreover, the file write does not occur within a request to the application. Therefore it is exempt from the 60 second limit that would otherwise apply and allows uploads of very large files.

https://cloud.google.com/appengine/docs/standard/php/googlestorage/user_upload#createuploadurl_options

But the python cloudstorage docs dont seem to have a reciprocal section

However, the old blobstore docs have a similarly name create_upload_url() where you specify gs_bucket_name:

https://cloud.google.com/appengine/docs/standard/python/refdocs/google.appengine.ext.blobstore.blobstore#google.appengine.ext.blobstore.blobstore.create_upload_url

So maybe that will do it

More info on the blobstore docs:

https://cloud.google.com/appengine/docs/standard/python/tools/webapp/blobstorehandlers#BlobstoreUploadHandler

Answer 2

Best way to upload big files to the Google Cloud is to do it directly to the Google Cloud Storage. It is possible to accomplish this task using two distinct methods:

1. Uploading parallel chunks (up to 32) and then composing them to one object. Using this approach, do not forget to delete temporary chunks.

2. Using resumable uploads, that will be automatically resumed when interrupted. Here is example code do upload video using javascript:

---