Google App Engine - using Google Cloud Storage in separate project, project has special characters in name

1
votes

I am trying to read and write from a GAE (Python) app to GCS bucket in a separate project. The other project:

  • has billing set up
  • has granted write-access to the GAE app's service account on the bucket it is attempting to write to.

However, I am not able to write to the bucket from my GAE app. The project (which I did not set up) has a colon (:) in the Project ID, and I get the error ValueError: Path should have format /bucket/filename....

I tried using the url-escaped version of the Project ID (i.e., in the first segment of the bucket path), and got the same error.

I also tried using the Project Number as the first segment of the path, but then it could not find the bucket: NotFoundError: Expect status [201] from Google Storage. But got status 404..

Is it possible to reference such a bucket? Do I need to use the x-goog-project-id header somehow?

EDIT: GCS adapter code

Note that the bucket is set reading from a config file, it ends up being something like Uploader('my:gcs-project/folder'). I am sure it is set correctly because of the error message I get.

import os.path
import cloudstorage as gcs

class Uploader():

  def __init__(self,bucket):
    self._bucket = bucket
    self._path   = ''
    self._meta   = {}
    self._type   = None

  def datasend(self, id):
    self._path = id
    return self

  def description(self, _):
    self._meta['x-goog-meta-desc'] = _
    return self

  def type(self, _):
    self._type = _
    return self

  def fullpath(self, filename):
    return "/".join([
             part for part in [
               "", self._bucket, self._path, os.path.basename(filename)
             ] if not (part is None)
           ])

  def __call__(self, file):
    self.upload(file)

  def upload(self, file):
    gcs_file = None
    try:
      gcs_file = gcs.open( self.fullpath(file.filename),
                           'w',
                           content_type=self._type,
                           options=self._meta
                         )
      for chunk in fbuffer(file.file):
        gcs_file.write(chunk)

    finally:
      if gcs_file:
        gcs_file.close()

def fbuffer(f, size=10240):
  while True:
    chunk = f.read(size)
    if not chunk: break
    yield chunk
1
pythongoogle-app-enginegoogle-cloud-storage
Can you show your code? I have a GAE app that successfully reads from a GCS bucket in another project with a colon in that project name. I suspect if I could see how your code differs from mine, I could help you. – akgill
OK, I added that above. I have a feeling it is a cloud config issue, but thanks for looking at it. – Eric G
Looking into it some more, I think I misunderstood - the project ID is not the first part of the bucket path, then? Instead you set this in the x-goog-project-id header? In which case it doesn't look like I can use the appengine GCS interface (which doesn't let you set that header), but instead the generic REST client (google-api-python-client) ? – Eric G
I didn't have to set any tags. GCS bucket names are all unique so no project needs to be specified. (I had forgotten this.) If you have set up permissions correctly for your app to access the bucket (which for me was just having the other app's service account with write permissions on the bucket), I think you'll be able to just access it. Try test_iterator = gcs.listbucket('/bucket/') and see if you get an iterator over the items in that bucket. – akgill
@Mario - the code above works as-is, provided the bucket name you pass in is simply the name of your bucket, not prefixed with your project name or id. – Eric G

1 Answers

1
votes

Try get your bucket with something like this:

bucket_name = os.environ.get('BUCKET_NAME', app_identity.get_default_gcs_bucket_name())
bucket = '/' + bucket_name
filename = bucket + '/' + "myfile.jpg"