google-fluentd and Stackdriver one entry behind on custom config

1
votes

I have a VM running on Google Cloud, and have installed logging via google-fluentd on it, according to the directions at https://cloud.google.com/logging/docs/agent/installation. I have set up a custom config for an application which outputs the logs to /var/log/myapp.log.

I created a configuration (as root, virtually a copy of tomcat.conf) called /etc/google-fluentd/config.d/myapp.conf which has the following:

<source>
  @type tail
  format multiline
  # Match the date at the beginning of each entry
  format_firstline /^(\d+\/\d+\/\d+\s\d+:\d+:\d+\s)/
  format1 /(?<message>.*)/
  path /var/log/myapp.log
  pos_file /var/lib/google-fluentd/pos/myapp-multiline.pos
  read_from_head true
  tag myapp
</source>

Entries are showing up in Stackdriver but it is getting into Stackdriver only after the next entry is put into the .log file and showing up as the wrong time (i.e., line-1 is getting line-2's timestamp).

Is there something that I'm missing in the .conf file that could be causing this?

1
fluentdstackdrivergoogle-cloud-logging

1 Answers

3
votes

From the documentation of the in_tail plugin:

With format_firstline, in_tail delays record emit until next format_firstline matched because in_tail can't judge multiline logs are ended or not without format_firstline trigger. If your regexps represent log pattern correctly like above Rails example, you may remove format_firstline for emitting records immediately.