Google Cloud - creating sink

Question

Im trying to export logs in to bigquery using sink from the cloud shell. I did the following steps: bq mk dataset

gcloud beta logging sinks create my-bq-sink \ bigquery.googleapis.com/projects/my-project/datasets/\ my_dataset --log-filter='resource.type="gce_instance"'

I created a service account for the sink and bind him to bigQuery.dataEditor and logging.logWriter

The problem is that if im not going to the console-> edit sink -> update sink im getting that my access to the dataset was denied. how can i solve that from cloud shell?

guillaume blaquiere guillaume blaquiere · Accepted Answer · 2020-05-21T19:32:19

Like in many products, creating a service is separate from the IAM authorization. For the logging sink, the "strange" decision from Google has been to generate a service account by the logging service and to send you the name of this service account in the command result

Created [https://logging.googleapis.com/v2/projects/My_PROJECT/sinks/test].
Please remember to grant `serviceAccount:[email protected]` the BigQuery Data Editor role on the dataset.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

Not very usable if you want to script something. So, add the parameter --format=json into the sink creation command and the result is the following

{
  "createTime": "2020-05-21T19:27:36.599050569Z",
  "destination": "bigquery.googleapis.com/projects/My_PROJECT/datasets/asset_eu",
  "filter": "resource.type=cloud_function",
  "name": "test",
  "updateTime": "2020-05-21T19:27:36.599050569Z",
  "writerIdentity": "serviceAccount:[email protected]"
}

Now you can get the writerIdentity and grant the role that you need on it. However, I repeat, this choice is strange for Google (and not consistant with other products) and I won't be surprised that this behavior change in the future.

Google Cloud - creating sink

1 Answers