Google Cloud Storage & Stackdriver Logging: how to read storage bucket logs with “gcloud”?

Question

I am trying to read the server-side logs for the buckets in my Google Cloud Storage project via the gcloud command-line program (to solve an error I get using the storage client). It does not seem like the logs are available in the Stackdriver Logging UI.

So, first question: are these logs available at all? If so, how do I access them?

It looks like it should be possible via gcloud:

Mortens-MacBook-Pro:~ skyfer$ gcloud logging resource-descriptors list | grep -i storage
gcs_bucket                    A Google Cloud Storage (GCS) bucket.                                                                       
project_id,bucket_name,location

But using gcloud logging read e.g. like the following doesn't work:

Mortens-MacBook-Pro:~ skyfer$ gcloud logging read "resource.type=gcs_bucket AND logName=projects/my-project/logs/ AND textPayload:StorageException" --limit 10 --format json
[]
ERROR: (gcloud.logging.read) INVALID_ARGUMENT: Name is missing the logs component. Expected the form projects/[PROJECT_ID]/logs/[ID]

So an important question is: what is the syntax of the ID? The documentation I have been able to find, e.g. LogEntry and Available Logs, doesn't seem to give the answer.

I have tried various formats and none have worked: they either return syntax errors or no results. I have also tried leaving out ID but that simply returns an empty list:

Mortens-MacBook-Pro:~ skyfer$ gcloud logging read "resource.type=gcs_bucket AND logName=projects/my-project/logs/my-bucket" --limit 10 --format json
[]

If I change the name of the bucket to a bucket that doesn't exist it doesn't seem to make a difference, indicating that the bucket name is not part of the id.

The bucket exists in Google Cloud Storage: verified both via the UI and gsutil:

Mortens-MacBook-Pro:~ skyfer$ gsutil ls gs://my-bucket/
gs://my-bucket/03ea8f19-8135-4101-a04a-aef3f19b0fdb/
gs://my-bucket/59e86a67-d035-4e4a-bc56-7c2da5e8c908/

Any help would be much appreciated.

Caner Caner · Accepted Answer · 2018-03-16T14:28:09

Yes the logs are not available in the stackdriver.

There is 3 types of logging for google cloud storage.

Access logs: They provide information for all of the requests made on a specified bucket and are created hourly

Daily storage logs: They provide information about the storage consumption of that bucket for the last day.

Cloud Audit Logging: This gives you access logs of API operations performed in Cloud Storage, you can find more details about it here.

Your log files will have a specific format and you can find more about it here.

Once you figure the log file you want to download, you can download it via:

gsutil cp gs://<bucket_name>/<file_name>

Google Cloud Storage & Stackdriver Logging: how to read storage bucket logs with “gcloud”?

2 Answers