stackdriver logging agent not showing logs read from a custom log file in stackdriver logging viewer on Google cloud platform

Question

I decided to post this question because, I have ran out of debugging ideas, just ideas are golden since I know it can be difficult to help debugging a virtual instance through here (debugging code is hard enough jaja). Anyway, I have created a virtual machine in Compute engine , I created a logs file that I populate, for example, with this command in a python script, let's call it logging.py:

import logging 
logging.basicConfig(filename= 'app.log'  , level = logging.INFO , format = ' %(asctime)s - %(name) - %(levelname)s - %(message)s')   

 logging.info('Some message ' + str(type(variable))) 

everytime I use python3 logging.py , the app.log is effectively populated. ( Logging.py and app.log are in the same directory the /home/username/ folder )

I want stackdriver to show this log in the logging viewer everytime it's written, so , I installed the stackdriver agent as follows, in the virtual machine command line:

    $ curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
    $ sudo bash install-logging-agent.sh

No errors that I see are delivered here, in fact, you can see here the messages obtained

Messags on the stackdriver viewer:

After this, I proceed to create a .conf file that I create in /etc/google-fluentd/config.d/app.conf

with this parameters

 <source>
 type tail
 format none
 path /home/username/app.log
 pos_file /var/lib/google-fluentd/pos/app.pos
 read_from_head true
 tag whatever-tag
</source>

After that is created, I launch sudo service google-fluentd restart.

Aftert I execute, python3 logging.py , no logs are added to stack drivers logging viewer.

So, where might Have I gone wrong?

Things I have tried/checked:

-Have more than 13 gygabytes of RAM available

-If I run logger "some message" on the command line, I effectively add a log with "some message" to the log viewer

-If I run

ps ax | grep fluentd

I obtain : 3033 ? Sl 0:09 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --no-supervisor 3309 pts/0 S+ 0:00 grep --color=auto fluentd

-Both my user, and the service account I use, have logger admin permission in IAM roles.

-This is the documentation I have based myself on:

https://cloud.google.com/logging/docs/agent/troubleshooting?hl=es-419 https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list?hl=es-419 https://cloud.google.com/logging/docs/agent/configuration?hl=es-419 https://medium.com/google-cloud/how-to-log-your-application-on-google-compute-engine-6600d81e70e3 https://cloud.google.com/logging/docs/agent/installation

-If I run sudo service google-fluentd status , the agent appears active.

-My instance hass access, to all the apis. It's an n1-standard-4 (4 vCPUs, 15 GB of memory) using ubuntu linux 18:04

So, what else can I check to debug this? I'm out of ideas here , hope I'm not being an idiot here :(

Answer 1

Based on my understanding, I think that you looking for the following fluentd resource types:

generic_node

“A generic node identifies a machine or other computational resource for which no more specific resource type is applicable. The label values must uniquely identify the node.”

generic_task

“A generic task identifies an application process for which no more specific resource is applicable, such as a process scheduled by a custom orchestration system. The label values must uniquely identify the task.”

The source of my information has been found here
This document explain how to send logs from your application in different ways:

• Cloud Logging API
• Cloud Logging Agent
• Generic fluentd

As you mentioned having installed fluentd, let me provide more focused documentation about Cloud Logging Agent. I also found some python Client Library documentation that you may be interested.

Finally, I found a nginx/apache use-case guide that you may use as reference.

Answer 2

For some reason, if I change the directory to which both the .conf file points, and the directory where the logg is to /var/logs/ , being the final path as /var/logs/app.logs, it does work correctly. Possibly there is a configuration issue, causing the logging agent to only capture logs in specific predetermined folders, or a permissions issue that stops it from working if the log is in the username directory.

I found this solution, however, by chance(random testing basically. ). Did not find anything in the main articles that are supposed to teach me how to configure the logging agent, that could point me in the right direction, being those articles this ones,

https://cloud.google.com/logging/docs/agent/troubleshooting?hl=es-419 https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list?hl=es-419 https://cloud.google.com/logging/docs/agent/configuration?hl=es-419 https://medium.com/google-cloud/how-to-log-your-application-on-google-compute-engine-6600d81e70e3 https://cloud.google.com/logging/docs/agent/installation

If I needed it to work in my username directory, it's not clear just by checking this articles how to do it,what configuration file I would need to change or where to start, so I recommend to google to improve that aspect of the docs.

This documentation you have sent https://docs.fluentd.org/quickstart is pretty interesting, maybe I can find the explanation there, thank you for your help.