google cloud storage client library - behind proxy - access bucket objects using python code?

Question

Centos Linux on vmware - gsutil is working but I am trying to download objects from google cloud storage using python code. Running below python code fails as I am behind a proxy server. I tried exporting http_proxy and https_proxy, also adding it via .boto (though i'm guessing only gsutil uses it). But none works.

I can't find any mention of proxy settings in the documentation as well.

from google.cloud import storage

storage_client = storage.Client()
bucket = storage_client.get_bucket('my-bucket')
blobs=bucket.list_blobs()

OSError: [Errno 101] Network is unreachable

*UPDATE 24-JULY-17 - RESOLVED *

re-installed google cloud storage library and my script works fine with HTTP_PROXY set in the env. Not sure about the root cause for initial troubles and I was not able to reproduce the error again unfortunately.

Tuxdude Tuxdude · Accepted Answer · 2017-07-19T02:31:05

Although, google-cloud python library does not have support for proxies directly, it honors HTTPS_PROXY environment variable if set.

Either:

export HTTPS_PROXY=https://mycustomproxy.example.com:12345
python your_python_script.py

Or:

export https_proxy=https://mycustomproxy.example.com:12345
python your_python_script.py

You could also set this directly within your python script too (preferably at the very beginning):

import os
os.environ['https_proxy'] = 'https://mycustomproxy.example.com:12345'

from google.cloud import storage
storage_client = storage.Client()
bucket = storage_client.get_bucket('my-bucket')
blobs=bucket.list_blobs()

BTW, https_proxy is supported in the urllib module and hence any libraries (like google-cloud here) using urllib can transparently use the proxies for the requests.

google cloud storage client library - behind proxy - access bucket objects using python code?

2 Answers