Confusion in various Client Libraries and apis to integrate App Engine with Cloud Storage

Question

'm bit new to the Google cloud platform and I'm trying to understand the app engine integration options with google cloud storage using client libraries / apis.

But I'm more confused now regarding the multiple choices of client libraries option that are there.

I tried looking for answers but I still have a few doubts. So I have posted specific questions/scenarios below. Also since few forums that I saw were bit old and so wanted to make sure that I get the answer as per current supported features in Google Cloud Platform as they keep changing/upgrading sometimes.

Could you please help me in clarifying my doubts below.

Apologies in advance because they might be very basic level questions since I'm new to this.

Is Google Cloud Storage CLient Library same as Google Cloud storage JSON client library? If yes, does Cloud storage Client library and JSON client library only limit to JSON apis of Cloud Storage? Or Cloud Storage Client library support XML apis too?

Lets say for example I want to perform resumable upload to cloud storage from app engine. Does it support both xml and json api using Google Cloud storage client library?

If not, does any other client libraries support xml and json apis for resumable upload from app engine?

Secondly, I saw an example in Cloud storage with Python, https://cloud.google.com/storage/docs/xml-api/gspythonlibrary

In this example, they have used boto library and google-cloud-oauth2-boto-plug.

But I also read in appengine docs that both Python and Java support oauth directly. So why do we need boto library and cloud-oauth2-boto-plugin? Is it because they are directly integrating python using xml apis with out using Cloud storage client library? And if they use Cloud storage client library, then python wont need to install boto library and cloud-oauth2-boto-plugin since it supports oauth by default?

Thanks.

Kiran

Answer 1

There are two public APIs for Google Cloud Storage, the XML API and the JSON API. The XML API looks and acts a lot Amazon's S3 API, and it's easy to use command line utilities like curl and wget to download and upload data. The JSON API looks like other Google Cloud APIs, and so if you're familiar with those APIs, the JSON API will be as you would expect.

There are a variety of client libraries:

Google API Client Libraries

The Google API Client Libraries are a set of libraries for accessing various Google APIs in a variety of programming languages. These all make use of GCS's JSON API.

Gcloud-* (<---- Use these)

There are also a set of libraries with names like gcloud-*, such as gcloud-java, gcloud-node, gcloud-python, etc. These libraries are the next generation of client libraries and remove a great deal of boilerplate code. They're quite nice, and they also use GCS's JSON API.

The Boto Library

Next, there's the boto library. The boto library is a third party Python library primarily aimed at supporting Amazon's AWS utilities. In addition to supporting AWS's S3, it also supports GCS via the XML API. Note that there's also a Boto3 library which does not support GCS. As the boto library is not aimed at Google and has no particular knowledge of Google's authentication mechanisms, there is an authentication plugin for boto called google-cloud-oauth2-boto-plug which provides that functionality.

GCS has a command-line utility called "gsutil". It's a great tool for interacting with GCS from the command line. It defaults to speaking to GCS via the JSON API, but it can be configured to use the XML API instead, and when using the XML API it makes use of the boto library to do so.

GCS App Engine Clients

In addition to all of this, there are additional, app engine only, GCS client libraries for Java and Python. These libraries use the XML API. Their main advantages over gcloud-java and gcloud-python is that they provide a basic local fake of GCS when developing with app engine's local development server and that they make it easy to save their upload/download state to be resumed later. If that's not important to you, I'd suggest just using gcloud-python instead.

A Note on Auth

When running code in the Google cloud, such as on GCE or on App Engine, Google makes it very easy to use OAuth to authentication as a service account automatically. The gcloud-* libraries take advantage of this. If you use them in App Engine, auth will just happen. You can read more about that in the gcloud-python auth guide.