Stackdriver Log Agent - Log Level Irrelevant with Google Cloud Logging Driver for Docker

6
votes

TL,DR; Log levels are ignored when making a Stackdriver logging API call using using a CloudLoggingHandler from a Docker container using the Google Cloud Logging driver.

Detail; The recommended way to get logs from a Docker container running on Google's Compute Engine is to use the Stackdriver Logging Agent:

It is a best practice to run the Stackdriver Logging agent on all your VM instances. The agent runs under both Linux and Windows. To install the Stackdriver Logging agent, see Installing the Logging Agent.

The following steps were completed successfully:

  • Ensure Compute Engine default service account has Editor and Logs Writer roles.
  • Ensure the VM instance has Cloud API access scope for Stackdriver Logging API (Full)
  • Install and start Stackdriver Logging Agent.

I then copied the example CloudLoggingHandler example from Google's Cloud Platform Python docs. 

import logging
import google.cloud.logging
from google.cloud.logging.handlers import CloudLoggingHandler

client = google.cloud.logging.Client()
handler = CloudLoggingHandler(client)

cloud_logger = logging.getLogger('cloudLogger')
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(handler)

cloud_logger.error('bad news error')
cloud_logger.warning('bad news warning')
cloud_logger.info('bad news info')

The Docker container is started with the Google Cloud Logging Driver flag (--log-driver=gcplogs):

sudo docker run --log-driver=gcplogs --name=server gcr.io/my-project/server:latest

This works, however all logs, irrespective of level are only visible in Stackdriver when viewing 'Any log level'. Strangely, the message itself contains the level:

2018-08-22 22:34:42.176 BST
ERROR:bad news error

2018-08-22 22:34:42.176 BST
WARNING:bad news warning

2018-08-22 22:34:42.176 BST
WARNING:bad news info

This makes it impossible to filter by level in the Stackdriver UI:

enter image description here

In the screenshot below, all icons on the LHS of every log entry show the level as Any:

enter image description here

1
dockergoogle-cloud-platformstackdrivergoogle-cloud-stackdriver
issuetracker.google.com/issues/35899495 – Martin Zeitler
Since you are receiving the logs, there is no issue with your configuration. The issue seems to be more with the Stackdriver Logging agent able to set the severity on the logs. The link provided by @MartinZeitler is from two years ago. I would suggest to create a new public issue where the GCP team can further investigate the issue with Stackdriver Logging here: issuetracker.google.com – Jason
Did you create an issue? – gelbander
@Jack Have you figured this out yet? – P_equals_NP_2021
For those stuck with this issue on GCE rather than GKE try switching up your instance to a Stackdriver Logging Agent supported OS rather than container optimized – P_equals_NP_2021

1 Answers

1
votes

From what I can tell, the CloudLoggingHandler is a standalone handler that sends logs to the Global log level. To integrate with gcplogs driver properly, try using the ContainerEngineHandler