Google App Engine, Cloud Storage can't read uploaded files in python

Question

I created the bucket, upload a file via

 gsutil cp -a public-read test.htm gs://BUCKETNAME

I can see the file in the Storage Browser and with gsutil ls

When I try to read it with the following code, I get

NotFoundError: Expect status [200] from Google Storage. But got status 404.

Code:

import logging
import os
import lib.cloudstorage as gcs
import webapp2

from google.appengine.api import app_identity


my_default_retry_params = gcs.RetryParams(initial_delay=0.2,
                                          max_delay=5.0,
                                          backoff_factor=2,
                                          max_retry_period=50)
gcs.set_default_retry_params(my_default_retry_params)

class MainPage(webapp2.RequestHandler):

    def get(self):
        bucket_name = os.environ.get('BUCKET_NAME',
        app_identity.get_default_gcs_bucket_name())

        self.response.headers['Content-Type'] = 'text/plain'
        self.response.write('Demo GCS Application running from Version: '
           + os.environ['CURRENT_VERSION_ID'] + '\n')
        self.response.write('Using bucket name: ' + bucket_name + '\n\n')

        bucket = '/' + bucket_name
        filename = bucket + '/test.htm'
        self.read_file(filename)


    def read_file(self, filename):
        self.response.write('Abbreviated file content (first line and last 1K):\n')

        gcs_file = gcs.open(filename)
        self.response.write(gcs_file)

app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

Strage thing happens, when I use the sample code from here (https://github.com/GoogleCloudPlatform/appengine-gcs-client/blob/master/python/demo/main.py) in my environment.

The code creates and writes a file on the same bucket, then reads it and deletes it. Works fine, but when I comment out the delete parts, I can not see the files in Storage Browser or with gsutil, but I can still read them with the code.

I checked my bucket permissions in Storage Browser and they seem fine, the service account of the app is bucket owner.

Any help is greatly appreciated.

I cant get my head around it. how is it possible, that I see the files in the browser and gsutil, but cant read them with the code, which is the same code as in the sample, which can read files it prior wrote. — aydunno
when I rund the code of this sample: cloud.google.com/appengine/docs/standard/python/blobstore/… it runs as expceted and the file seems to be written, but I cant see it with gsutil or the code form github. HONESTLY, THREE WAYS TO WRITE THE FILES AND THEY SHOW UP NOWHERE EXPECT IN THEIR OWN SPACE / RUNTIME / CACHE? — aydunno
I tried your code and wasn't able to replicate your issue. I did have to go and add my service account to the permissions, but was then able to upload via either gsutil or the storage browser and read the files fine. I was able to also see the files fine in the storage browser. Have you taken a look at the logs viewer console.cloud.google.com/logs/viewer ? Also, you haven't explicitly stated if you are testing locally or deployed? — BrettJ
Thanks a bunch Brett. I tested locally and deployed, the logs state what I wrote in my post. Its crazy...I added the service accoun in the permissions, too. I will double check again and/or start over. what permissons did you assign exactlly? legacy > bucket owner? — aydunno
Just wanting to confirm that you are removing the true bucket name and replacing with BUCKETNAME for StackOverflow. — bscott

aydunno aydunno · Accepted Answer · 2018-02-22T09:39:39

After revisiting this, due to recent demand I figured it out. gsutil has a bug, I read about its somewhere at github. When I upload the file via storage browser or the python client libraries it works.

Google App Engine, Cloud Storage can't read uploaded files in python

1 Answers