How do to use fluentD to parse multi-line docker logs?

4
votes

I'm trying to aggregate my docker-compose logging environment using the fluentD driver, fluent, elastic, kibana.

The logs forward properly but the nested JSON objects each get recorded as a single event (see image).

enter image description here

Is the proper method here to structure the data using custom regex?

I have the following in my fluentd.conf

<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>

<match docker.**>
  type elasticsearch
  logstash_format true
  logstash_prefix logstash
  host elasticsearch
  port 9200
  flush_interval 5s
</match>
1
elasticsearchdockerkibanafluentd

1 Answers

0
votes

Tkwon123 - Am i correct that you want to write JSON logs in your docker containers and have each JSON object show up as 1 log in kibana?

If I am understanding you correctly it seems like you just need to make sure your JSON logs are being printed with only an ending carriage return. In looking at your screen shot it sort of looks like you may be pretty printing the JSON object which is breaking it apart over multiple lines.

Hope i understood your question.