Parse different formats using fluentd from same source given different tag?

Question

I'm using fluentd in a docker-compose file, where i want it to parse the log output of an apache container as well as other containers with a custom format.

In order to differentiate the formats, I'm planning to set tags in docker-compose like this:

      logging:
        driver: "fluentd"
        options:
          tag: "apache2"

So fluentd should be able to use different formats based on the tag. But how can I configure fluentd to do this?

The documentation says one should put this in the source section (which I can't do because I need two different formats):

  <parse>
    @type apache2
  </parse>

My very basic source looks like this:

<source>
  @type forward                                                                                         
  port 24224
  bind 0.0.0.0
</source>

Is it possible to use fluentd routing to use two different formats for data coming from the same source with different tags?

Max Lobur Max Lobur · Accepted Answer · 2020-09-01T16:56:42

Yes it's possible:

# 1. Omit parsing at the source
<source>
  @type forward                                                                                         
  port 24224
  bind 0.0.0.0
</source>

# 2. Write a dedicated filter for each format you want
<filter docker.apache2> # Check your exact produced tag, depends of versions. Just guessing here.
  @type parser
  key_name log
  <parse>
    @type apache2
  </parse>
</filter>

<filter docker.backend>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

# 3. Match and store all
<match **>
  @type s3 
  ...
</match>

Parse different formats using fluentd from same source given different tag?

1 Answers