Logging with Fluentd - why is the output of json log file appearing as textpayload (not jsonpayload)?

2
votes

I am new to Fluentd. I'm using stackdriver in GKE and I am customizing the Fluentd configuration in GKE to make some changes to the logs.

In my configuration file for the logs of my containers I have: 

<source>
  type tail
  format json
  time_key time
  path /var/log/containers/*.log
  pos_file /var/log/gcp-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%N%Z
  tag reform.*
  read_from_head true
</source>

The logs of some containers are json objests, but I see their output as textpayload (when I enable the built-in Fluentd on GKE they appear as jsonpayload).

I don't understand what could cause this. I would appreciate any advice.

1
jsonloggingfluentdstackdrivergoogle-kubernetes-engine

1 Answers

2
votes

I'm not sure if this answer will cover your case, but it may save few hours of investigation to someone like it could have to me.

Although format parameter is now deprecated and replaced with <parse>, it does support json parsing.

The crucial thing here is JSON object structure.
First nginx access log example that I've found do NOT work with Stackdriver without modification: 

log_format json_combined escape=json '{ "time_local": "$time_local", '
 '"remote_addr": "$remote_addr", '
 '"remote_user": "$remote_user", '
 '"request": "$request", '
 '"status": "$status", '
 '"body_bytes_sent": "$body_bytes_sent", '
 '"request_time": "$request_time", '
 '"http_referrer": "$http_referer", '
 '"http_user_agent": "$http_user_agent" }';

And this doesn't work because JSON is single level and Stackdriver requires it to be nested at least once under a single key. (At least in this example with fluentd agent, not sure for other cases)

This can be any key like accessLog in WORKING example below. 

log_format json_combined escape=json '{ "accessLog": { "time_local": "$time_local", '
 '"remote_addr": "$remote_addr", '
 '"remote_user": "$remote_user", '
 '"request": "$request", '
 '"status": "$status", '
 '"body_bytes_sent": "$body_bytes_sent", '
 '"request_time": "$request_time", '
 '"http_referrer": "$http_referer", '
 '"http_user_agent": "$http_user_agent" } }';

If using httpRequest on the first JSON level be warned that it has different purpose.