JSON format for Stackdriver logging in Google Kubernetes Engine

Question

I'm currently looking for simplest possible JSON log messages that would simply write a severity and a message to Stackdriver logging from a container that is run in Kubernetes Engine and is using the managed Fluentd daemon.

Basically I'm writing single line JSON entries as follows.

{"severity": "DEBUG", "message": "I'm a debug entry"}
{"severity": "ERROR", "message": "I'm an error entry"}

These end up in Stackdriver logging with following results.

Severity is always INFO
There's JSON payload in the log entry, and the only content is the message, i.e. severity does not go there.

My conclusion is that Fluentd recognizes log row as JSON, but what I don't understand is that how the severity is not set into log entries correctly. Am I e.g. missing some mandatory fields that need to be in place?

Is there any reason why you are not using a client library that would do the correct formatting automatically? — DevopsTux
@JordiMiralles Said client libraries can be unreasonably large, when all you wanted was a single struct definition. — Nelo Mitranim

DevopsTux DevopsTux · Accepted Answer · 2018-03-20T16:42:57

From the information you provided I guess fluentd is passing your whole JSON as as a jsonpayload as a logEntry and providing the logName, resource type and the rest of required information from the environment variables.

In the end what Stackdriver is receiving must look something like this:

{
 "logName": "projects/[YOUR PROJECT ID]/logs/[KUBERNETES LOG]",
 "entries": [
  {
   "jsonPayload": {
    "message": "I'm an ERROR entry",
    "severity": "ERROR"
   },
   "resource": {
    "labels": {
     "project_id": "[YOUR PROJECT ID]",
     "instance_id": "[WHATEVER]",
     "zone": "[YOUR ZONE]"
    },
    "type": "gce_instance"
   }
  }
 ]
}

So you are actually getting the content of the JSON payload on Stackdriver, but as the severity is defined either outside the JSON payload or, if you want to do it inside you'll have to use "severity": enum([NUMERICAL VALUE])

The numerical values of each log level are:

Enums
DEFAULT (0) The log entry has no assigned severity level.
DEBUG (100) Debug or trace information.
INFO (200) Routine information, such as ongoing status or performance.
NOTICE (300) Normal but significant events, such as start up, shut down, or a configuration change.
WARNING (400) Warning events might cause problems.
ERROR (500) Error events are likely to cause problems.
CRITICAL (600) Critical events cause more severe problems or outages.
ALERT (700) A person must take an action immediately.
EMERGENCY (800) One or more systems are unusable.

So, including the field "severity": enum(500) should log the entry as an ERROR instead to fallback to the default INFO.

JSON format for Stackdriver logging in Google Kubernetes Engine

3 Answers