how to combine multiline docker log into one fluentd event

Question

Hope someone could help me because this problem has confused me for several days...

I'm using fluentd to centralize docker container's logs. so I set docker run options:

log-driver=fluentd
log-opt=fluentd-address=fluentd.marathon.mesos:24224
log-opt=tag=server
log-opt=fluentd-async-connect=true

and received json objects like

{container_id:xxx, container_name:xxx, log:xxx, ...}

for example a java exception stack trace will be like this:

{log:exampleException:error, ...}
{log:\tat java.xxx, ...}
{log:\tat java.xxx, ...}

Now my question is: how can I combine them together? Like multline in logstash.

The correct answer I'm expecting is:

{log:exampleException:error\n\tat java.xxx\n\tat java.xxx, ...}

My fluentd input is in_forward not in_tail, so from the document I can't use the multiline plugin built in fluentd core.

But because my log received from docker doesn't contain '\n' at the end of each line, this plugin combined all my log into only one line!

e.g. I use the word "Exception" to mark the firstline, but the answer I get is:

{log:Exception:error\tat java.xxx\tat java.xxx, ...}

It treated my docker log as a single line and seperated them by word "Exception"

I have tried to add a '/n' to every log by filter record_transformer, but the filter changed my '\n' into '\\n'automatically, so it also fails

I also find a plugin https://github.com/okkez/fluent-plugin-concat but have problem installing this plugin

Recently I haven't found any method

Night Night · Accepted Answer · 2016-06-15T09:10:17

My answer is installing the multiline-parser plugin and enable ruby to add the '\n'

<filter server.**>
  type record_transformer
  enable_ruby
  <record>
    log ${record["log"]+"\n"}
  </record>
</filter>