client.get_bucket() fails, but only from cloud dataflow (compute engine)

Question

Application has been working normally, now on a re-deploy, google storage is giving strange errors.

MissingSchema: Invalid URL 'None/storage/v1/b/my-bucket-name?projection=noAcl': No schema supplied. Perhaps you meant http://None/storage/v1/b/my-bucket-name?projection=noAcl?

  File "/usr/local/lib/python2.7/dist-packages/lib/file_store.py", line 11, in __init__
    self.bucket = self.client.get_bucket(parts[0])
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/client.py", line 301, in get_bucket
    bucket.reload(client=self)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/_helpers.py", line 130, in reload
    _target_object=self,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 392, in api_request
    target_object=_target_object,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 269, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_http.py", line 298, in _do_request
    return self.http.request(url=url, method=method, headers=headers, data=data)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py", line 208, in request
    method, url, data=data, headers=request_headers, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 313, in prepare
    self.prepare_url(url, params)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 387, in prepare_url
    raise MissingSchema(error)
MissingSchema: Invalid URL 'None/storage/v1/b/my-bucket-name?projection=noAcl': No schema supplied. Perhaps you meant http://None/storage/v1/b/my-bucket-name?projection=noAcl? [while running 'generatedPtransform-51']

The code causing the error, I can run this locally using the same service account and it works, no error. I am using $env:GOOGLE_APPLICATION_CREDENTIALS to export my service account credentials at deploy time. All other services are working normally.

# My test is:
# fs = FileStore("gs://my-bucket-name/models/", "development", "general")

class FileStore():
    # modelPath - must be a gs:// style google storage resource path containing everything but the file extension
    def __init__(self, modelPath, env, modelName):
        from google.cloud import storage

        parts = modelPath[5:].split('/', 1)
        self.client = storage.Client()
        self.bucket = self.client.get_bucket(parts[0]) # <- error here

Why would google core client fail to build a URL? Based on 'None/storage/v1/b/my-bucket-name?projection=noAcl', the missing part of the URL should be something like "https://www.googleapis.com".

tgreiser tgreiser · Accepted Answer · 2019-09-06T23:21:31

This error is apparently caused by a mismatch between google_cloud_storage and google_cloud_core. I had specified google_cloud_core >= 1.0.3 in my setup.py but when I looked on the docker image on the compute VM I found it had an earlier version.

After rebuilding my venv from setup.py I had to also run:

C:\Python27\python.exe -m pipenv install google-cloud-core>=1.0.3 --skip-lock

Then I was able to deploy and the application started working again.

client.get_bucket() fails, but only from cloud dataflow (compute engine)

1 Answers