How to access files on Google Cloud Storage using Python

2
votes

I have files in a Storage Bucket that I can't access from python using the Cloud Shell. I created this simple example:

open('gs://iwc/panda.jpg')

which results in this:

jklieb@cloudshell:~$ python demo.py                                                                                                                                                                          
Traceback (most recent call last):
  File "demo.py", line 1, in <module>
    open('gs://iwc/panda.jpg')
IOError: [Errno 2] No such file or directory: 'gs://iwc/panda.jpg'

But the file is visible when using gsutil:

jklieb@cloudshell:~$ gsutil ls gs://iwc

gs://iwc/panda.jpg

I created a separate virtual machine instance (not the temporary one associated with Cloud Shell) and had a similar result.

I have also provisioned a TPU using ctpu and was similarly unable to access files in another storage bucket. The documentation on ctpu says that the permissions should be properly set up to allow access to storage buckets.

Have I made some simple mistake that is too obvious for me to see?

1
pythongoogle-cloud-storagegoogle-compute-engine

1 Answers

1
votes

The solution is to use tensorflow.gfile.Open() to access the files:

jklieb@iwildcam2018:~$ python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow.gfile as gf
>>> gf.Open('gs://iwc/panda.jpg')
<tensorflow.python.platform.gfile.GFile object at 0x7f03e8b15f28>

This works in the cloud shell and on other virtual machines I created. tensorflow.gfile.Glob() may be used to get a list of files matching a pattern.