Google App Engine: store uploaded file in Google Cloud Storage

Question

I have managed to create a file in GCS in the post method of a webapp2 handler of a Google App Engine application. I can't see how to copy the content of a posted file in the file created in GCS. Here is my code

 inFile =  self.request.POST.multi['file'].file
 gcs_file = gcs.open(filename,
                    'w',
                    content_type='text/plain',
                    options={'x-goog-meta-foo': 'foo',
                             'x-goog-meta-bar': 'bar'},
                    retry_params=write_retry_params)
  while 1:
        line = inFile.readline()
        if not line: break
        gcs_file.write(line)
  gcs_file.close()

At the end of the process the file in GCS is 0 byte

UPDATE There is a reason why I am not using the blobstore. When using the Blobstore you have to create an url and sand it back to the client. it is the client that performs the actual upload. INSTEAD I need to encrypt the file on the server before to put it in the GCS. Thus I need to receive the file from the client, encrypt it on the server and store it in the GCS.

Is there a reason you're not writing the whole file at once? Also, have you verified, maybe with logging, that you've properly received the file with the content length you expect from the POST? — Zach Young
Ah, I remember you from your prev question. You seem dead set on uploading to your webapp2 handler. Just stop. You are doing this wrong. USE THE BLOBSTORE. Yes. You use the blobstore to get the file to GCS. You really should just read up the whole blobstore overview. I'm not sure you are going to get any help here if you don't do the basic steps. — user2771609
@Zachary I have already checked the file is uploaded by verifying self.request.POST.multi['file'].filename actually returns the expected string. I hope this means the content is uploaded as well. I have also tried gcs_file.write(inFile.read()) but still get 0 bytes — Andrea Sindico
@Zachary I have verified the byte is correctly uploaded as the uploaded size is correct. But there something wrong with that loop as it enters only once (and the readline returns null). How should I read the file contents? — Andrea Sindico
@user2771609: "I'm not sure you are going to get any help here if you don't do the basic steps."... Google provides this API, so, it's valid. Even if Sindico didn't really need to bypass Blobstore and just wanted to do it this way, this is still a solution that's well enough (for GAE) documented. — Zach Young

Zach Young Zach Young · Accepted Answer · 2015-05-07T16:27:37

I've successfully POST'ed a file to GCS with the following method:

def post(self):
    write_retry_params = gcs.RetryParams(backoff_factor=1.1)
    filename = '/{MY-BUCKET-NAME}/static.txt'

    gcs_file = gcs.open(
        filename,
        'w',
        content_type='text/plain',
        options={'x-goog-meta-foo': 'foo',
                 'x-goog-meta-bar': 'bar'},
        retry_params=write_retry_params)

    inFile = self.request.POST.multi['file'].file
    while 1:
        line = inFile.readline()
        if not line:
            break
        gcs_file.write(line)
        logging.info('Wrote line: {}'.format(line))

    gcs_file.close()

Here's the little log message from Console:

I 09:20:32.979 2015-05-07  200      84 B   1.13s /static
    76.176.106.172 - - [07/May/2015:09:20:32 -0700] "POST /static HTTP/1.1" 200 84 - "curl/7.37.1" "{MY-APP}" ms=1131 cpu_ms=1355 cpm_usd=0.000009 loading_request=1 instance=00c61b117cee89e66d34a42c5bbe3cf2b0bb06b5 app_engine_release=1.9.20
I 09:20:32.832 Wrote line: stack
I 09:20:32.833 Wrote line: overflow

Here's the test.txt file I uploaded:

stack
overflow

And the cURL command I used:

curl -F"file=@/Users/{name}/test.txt" http://{MY-APP}/videostatic

If you're still getting 0 bytes from readline() or read() I'm going to have to assume that your client is not sending the correct multipart message.

Google App Engine: store uploaded file in Google Cloud Storage

2 Answers