How to logging python script log into Google stackdriver logging running on a Google Cloud VM

Question

I am using a Google Cloud virtual machine to run my python scripts scheduled on a cron, I am looking for some way to check the script logs and also to add my script running log into google stackdriver logging

is there anyway to add python script log into google stackdriver logging rather appending into log file in compute engine .What are the usual approaches for such things? Please suggest.

• • • • • /usr/bin/python /home/parida_srikanta/test.py >> logs.log 2>&1

BR/ SRIKANTA

Answer 1

A possible solution would be to make a script which reads the logs.log file and writes the logs to Stackdirver assuming your VM has the necessary permissions.

When using Compute Engine VM instances, add the cloud-platform access scope to each instance. When creating a new instance through the Google Cloud Console, you can do this in the Identity and API access section of the Create Instance panel. Use the Compute Engine default service account or another service account of your choice, and select Allow full access to all Cloud APIs in the Identity and API access section. Whichever service account you select, ensure that it has been granted the Logs Writer role in the IAM & admin section of the Cloud Console.

Setting Up Cloud Logging for Python

For example the following script:

 import google.cloud.logging
 import logging


 client = google.cloud.logging.Client()

 client.setup_logging()

 text = 'Hello, world!'

 logging.warning(text)

You can find the logs under the python log in the Global resource type.

Answer 2

You can use Cloud Logging agent with fluentd, in this way you don't have to change your script, and you're able to keep local log files on your VMs.

Main lines :

• Setup logging agent on your VM (manually or via startup-script)
• Setup fluentd conf to create a dedicated log for your scripts
• Add logging
• Retrieve your logs via Cloud Logging Viewer

See Official documentation to install Cloud Logging Agent, and How to configure it.

Main steps :

Install agent on your VM:

curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh

Add a dedicated configuration for your new local fluentd log inside /etc/google-fluentd/config.d/ :

<source>
    @type tail
    # Format 'none' indicates the log is unstructured (text).
    format none

    # The path of the log file.
    path /tmp/your-script-log.log

    # The path of the position file that records where in the log file
    # we have processed already. This is useful when the agent
    # restarts.
    pos_file /var/lib/google-fluentd/pos/your-script-log.pos

    read_from_head true

    # The log tag for this log input.
    tag your-script-log
</source>

Restart the agent

sudo service google-fluentd restart

Write to your log file : echo 'Test' >> /tmp/your-script-log.log

You will retrieve your log inside Cloud Logging Viewer

See also my answer to a different question but with common objective.