How to authorize access to Google Cloud Storage API in PHP, without the client library?

Question

I want to access the Google Cloud Storage API directly with my own PHP code, because Google's API library is slow, but I can't figure out how to get authorization to work. See code below.

I am trying to list the files in a "directory" within a Google Cloud Storage bucket (i.e. get all the files with a certain prefix), in PHP code running on App Engine Standard. I found that the native PHP functions (e.g. scandir()) which are implemented to do this on AppEngine are horrifically slow (> 1 second). I'm not talking about huge directories either, maybe 10 files.

I also tried using the Google APIs Client Library, specifically the Objects method, but found it to still be pretty slow, though not quite as bad, like around 0.5 seconds. The strange thing is, when I timed it, the actual "Objects()" call is fast, it's entering the loop ("foreach ($objects as $object)" - but just the first iteration) that's slow! So I think the library may be doing unnecessary work in processing the response. I verified this in "Try this API", looking at the request my browser makes, which takes < 100ms.

So, I want to make and process the API request directly, rather than using Google's slow library. This is the barebones function I wrote:

function get_files_from_gcs($dir, $data_storage_bucket) {
    $params = array(
        'delimiter' => '/',
        'prefix' => $dir,
        'fields' => 'items/name',);
    $query = http_build_query($params);
    $url = "https://www.googleapis.com/storage/v1/b/" . $data_storage_bucket . "/o?" . $query;
    log_error('Curl URL:'  . $url);
    //  Initiate curl
    $ch = curl_init();
    // Will return the response, if false it print the response
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    // Set the url
    curl_setopt($ch, CURLOPT_URL, $url);
    // Execute
    $result = curl_exec($ch);
    // Closing
    curl_close($ch);

    $data = json_decode($result, true);
    $filenames = array();
    foreach ($data as $item) {
        if (array_key_exists('name', $item)) {
            $filenames[] = $item['name'];
        }
    }

    return $filenames;
}

However, I get an error message in the response: "Anonymous users does not have storage.objects.list access to bucket ."

I have set public-read ACLs for my bucket like so:

gsutil defacl ch -u AllUsers:R gs://<MY BUCKET>
gsutil -m acl -r set public-read gs://<MY BUCKET>

And I also tried setting the 'project' URL param in my request, and/or creating an API Key and setting the 'key' param. No dice.

So I think I need to get an OAuth token (which I think is kind of silly - my App Engine Standard instance should be able to make requests to my GCS within the same project without any special authorization... but oh well). But honestly, I am pretty lost on how to do this. I've started looking through the Google Client API code, but I can't tell where this is actually happening. Any help would be much appreciated.

You realize that Google APIs are rest APIs and there by run though a webserver. You also need to remember that they all have flood protection. If you go any faster that that your going to start getting 400 and 500 errors. — DaImTo
Right, in my question I specify that I tried the JSON API directly and found that it was significantly lower latency for a single request than Google's library. I don't need to make a lot of requests all at once, I just want to be able to make one with <500ms of latency. — Andrew

pinoyyid pinoyyid · Accepted Answer · 2017-03-30T08:42:49

If I understand your question correctly, you have a working solution, but it's slow.

On that basis, I would suggest you use a hybrid approach. If you have auth working currently using the library, then keep that code without change. They key bit from the php documentation is:-

$token = $client->fetchAccessTokenWithAuthCode($_GET['code']);
$client->setAccessToken($token);

This is where you get the magical Access Token. Once you have that, you are free to construct your own http/rest calls. All you need to do is instantiate that Access Token into an http Authorization header, thus

Authorization: "Bearer 4979e799a7979b79etc..."

How to authorize access to Google Cloud Storage API in PHP, without the client library?

1 Answers