Hope someone could help me because this problem has confused me for several days...
I'm using fluentd to centralize docker container's logs. so I set docker run options:
log-driver=fluentd
log-opt=fluentd-address=fluentd.marathon.mesos:24224
log-opt=tag=server
log-opt=fluentd-async-connect=true
and received json objects like
{container_id:xxx, container_name:xxx, log:xxx, ...}
for example a java exception stack trace will be like this:
{log:exampleException:error, ...}
{log:\tat java.xxx, ...}
{log:\tat java.xxx, ...}
Now my question is: how can I combine them together? Like multline in logstash.
The correct answer I'm expecting is:
{log:exampleException:error\n\tat java.xxx\n\tat java.xxx, ...}
My fluentd input is in_forward not in_tail, so from the document I can't use the multiline plugin built in fluentd core.
Then I found this plugin https://github.com/quick-sort/fluent-plugin-multiline-parser
But because my log received from docker doesn't contain '\n' at the end of each line, this plugin combined all my log into only one line!
e.g. I use the word "Exception" to mark the firstline, but the answer I get is:
{log:Exception:error\tat java.xxx\tat java.xxx, ...}
It treated my docker log as a single line and seperated them by word "Exception"
I have tried to add a '/n' to every log by filter record_transformer, but the filter changed my '\n' into '\\n'automatically, so it also fails
I also find a plugin https://github.com/okkez/fluent-plugin-concat but have problem installing this plugin
Recently I haven't found any method