List Google Cloud Storage buckets using Python

2
votes

I'm following this tutorial: https://developers.google.com/storage/docs/gspythonlibrary and getting couple of errors while trying to list my buckets.

I've downloaded gsutil and added it to my PYTHONPATH which looks like this:

/home/nicolasalvo/tools/gsutil/third_party/boto:/home/nicolasalvo/tools/gsutil

I've also executed:

pip install -U oauth2client

The code I'm trying to run is:

import StringIO
import os
import shutil
import tempfile
import time
from gslib.third_party.oauth2_plugin import oauth2_plugin

import boto

# URI scheme for Google Cloud Storage.
GOOGLE_STORAGE = 'gs'
# URI scheme for accessing local files.
LOCAL_FILE = 'file'

uri = boto.storage_uri('', GOOGLE_STORAGE)
for bucket in uri.get_all_buckets():
  print bucket.name

The first error I've got is:

File "<stdin>", line 1, in <module>
File "/home/nicolasalvo/tools/gsutil/gslib/third_party/oauth2_plugin/oauth2_plugin.py", line 3, in <module>
import oauth2_client
File "/home/nicolasalvo/tools/gsutil/gslib/third_party/oauth2_plugin/oauth2_client.py", line 33, in <module>
import socks

Which I've fixed manually changing:

import socks

to be

import httplib2.socks

Now the error I'm facing is:

File "/home/nicolasalvo/tools/gsutil/third_party/boto/boto/connection.py", line 876, in _mexe
request.authorize(connection=self)
File "/home/nicolasalvo/tools/gsutil/third_party/boto/boto/connection.py", line 377, in authorize
connection._auth_handler.add_auth(self, **kwargs)
File "/home/nicolasalvo/tools/gsutil/gslib/third_party/oauth2_plugin/oauth2_plugin.py", line 22, in add_auth
self.oauth2_client.GetAuthorizationHeader()
File "/home/nicolasalvo/tools/gsutil/gslib/third_party/oauth2_plugin/oauth2_client.py", line 325, in GetAuthorizationHeader
return 'Bearer %s' % self.GetAccessToken().token
File "/home/nicolasalvo/tools/gsutil/gslib/third_party/oauth2_plugin/oauth2_client.py", line 288, in GetAccessToken
token_exchange_lock.acquire()
NameError: global name 'token_exchange_lock' is not defined

I've tried to declare the global object before using it, but it hasn't helped.

Also I would expect that I should be able to use Google provided libraries for Cloud Storage without requiring manual fixes.

I'm running Python 2.7.3

Any help is greatly appreciated

2
pythongoogle-cloud-storagebotogsutil
Any reason you don't want to use google-api-python-client? – jterrace
The JSON API has is marked as experimental, so I though the XML APIs were more stable. I've tried using the google-api-client and it works, but I will still be interested in solving the problem with the other API as well.. – ROOTto

2 Answers

6
votes

This worked for me:

import threading
from gslib.third_party.oauth2_plugin import oauth2_client
oauth2_client.token_exchange_lock = threading.Lock()
4
votes
import StringIO
import os
import shutil
import tempfile
import time
import threading
import boto

from gslib.third_party.oauth2_plugin import oauth2_plugin
from gslib.third_party.oauth2_plugin import oauth2_client

oauth2_client.token_exchange_lock = threading.Lock()

# URI scheme for Google Cloud Storage.
GOOGLE_STORAGE = 'gs'

# URI scheme for accessing local files.
LOCAL_FILE = 'file'

project_id = 'abc.com:abc'

uri = boto.storage_uri('', GOOGLE_STORAGE)

header_values = {"x-goog-project-id": project_id}

for bucket in uri.get_all_buckets(headers=header_values):
    print bucket.name

Yes it works!!