fluentd aggregator not getting logs from forwarder, is the config correct?

Question

I am currently trying to setup a system of forwarder and aggregator instances of fluentd. My forwarder config is ->

<source>
   @type tail
  format json
  path /app/node-apps/something-fe/logs/itt-logs.log
  tag td.pm2.access
</source>

## File output
## match tag=local.** and write to file
<match td.pm2.*>
  @type file
  path /opt/td-agent/forward/log/fluentd.log
</match>

## Forwarding
## match tag=system.** and forward to another td-agent server
<match td.pm2.*>
  @type forward
  host <hostname>.bc

Doing this i can see that t forwarder is outputting log files in the forwarding location here : /opt/td-agent/forward/log/fluentd.log All good so far!!! But when i try to import this in the aggregator via the match-forward syntaxes above i do not get anything in the aggregator machines. Please find teh aggregator config for fluentd here that i am using -->

<source>
  @type forward
      port 24224
</source>

<match td.pm2.*>
  type copy
    <store>
    @type file
    path /opt/td-agent/log/forward.log
  </store>
  <store>
    type elasticsearch
    host <aggreatorhost>.bc
    port 9200
    logstash_format true
    flush_interval 10s
  </store>
</match>

I am trying to use a store to copy the logs over there and also forward them to elasticsearch. Forgetting elasticsearch altogether, it seems that even the logs are not getting populated from the forwarder to the aggregator. Am i doing something wrong? The aggregator logs say that its listening on all addresses on port 24224.

ted-k42 ted-k42 · Accepted Answer · 2017-01-24T05:10:42

On your forwarder you have two identical match patterns and only the first match is being executed (the config is run top to bottom). The logs are being written to the file system (/opt/td-agent/forward/log/fluentd.log) but not forwarded to the aggregator.

You've actually used the correct copy syntax on the aggregator which you should copy into your forwarder and replace the elasticsearch with the @forward config to the aggregator

<match td.pm2.*>
  type copy
    <store>
    @type file
    path /opt/td-agent/log/forward.log
  </store>
  <store>
    @type forward
    host <hostname>.bc
  </store>
</match>

Further reading: http://docs.fluentd.org/articles/out_copy

fluentd aggregator not getting logs from forwarder, is the config correct?

2 Answers