How to change the StackDriver logging agent configuration in Compute Engine VMs?

Question

I am creating for Log exports for logs generated in Google Stackdriver Logging for all the Compute Engine VMs and Clusters present in a project.
As we know exports destination can only be Cloud Storage, Cloud Pub/Sub, BigQuery. Hence I am not able to differentiate the logs generated by multiple VMs.
For Example: There are a number of VMs present and all of them generate syslog, and have one cloud bucket as their destination, the sinks for all the VMs will export the logs in a directory named syslog in that same cloud bucket.

To overcome above problem, I followed a way in which we configure the StackDriver Logging agent and change the logName tag. See this for more details.
Using the above approach I am able to change the tag of the logName from syslog to [instance-name]-syslog and hence sinks export the syslog from that GCE instance to one folder and so on.

I am changing the tag in the /etc/google-fluentd/config.d/[APPLICATIOON-NAME].conf files manually as of now. For example, here is the /etc/google-fluentd/config.d/syslog.conf file:

<source>
  @type tail

  # Parse the timestamp, but still collect the entire line as 'message'
  format syslog

  path /var/log/syslog
  pos_file /var/lib/google-fluentd/pos/syslog.pos
  read_from_head true

  tag syslog

</source>

Now I am replacing the syslog above with [instance-name]-syslog manually by doing ssh in all the VMs. I am looking for a way to automate the above thing. What would be the most suitable way to do this? (P.S. All the VMs are being spin up using Terraform)

Does a record written to the log not contain the CE instance that generated the log? Could one not write all the records to one GCS object and then filter that object by the desired CE instance? What about output all the Log Export data to Pub/Sub and having a cloud function write the records to different GCS buckets? What about sending the data to BigQuery and hosting it there as a warehouse and simply doing a SQL query to get the data when / if needed? — Kolban
We would not prefer Pub/Sub or Bigquery to get in picture. And yes the logs contain the ID and name of the instance, and filtering the logs and putting them to different folders is the last solution we would consider to implement because for doing that, there would be an additional server/Instance/cluster would be required to run the script periodically. Is there other way to configure the log names and get the work done? — amit yadav
One can define an arbitrary starup script that is executed when each Compute Engine starts. See cloud.google.com/compute/docs/startupscript ... if you can script the desired effect to your Linux environment then you could define that to happen at machine boot. — Kolban

Calvin Zhou Calvin Zhou · Accepted Answer · 2019-04-01T10:29:14

First of all, we uploaded our logs to StackDriver so I don't face a same issue as you.

Back to you question, I think this is out of Terraform as it is only doing server creation. I would suggest 2 options:

Use some CM tools like Ansible to inject a filename based on the new server created
If you don't have a CM in place, you can create a customized image with a startup script, the script will check if the conf exists, if not, create a conf based on hostname in path /etc/google-fluentd/config.d/

Apparently, the 2 approaches are targeting the same thing and #1 is a long term one

How to change the StackDriver logging agent configuration in Compute Engine VMs?

1 Answers