How to enable fluentd pluggins

Question

I am using EFK.

Could someone help here on how to enable fluentd plugins.

I have installed the following 3 plugins as part of my work.

gem install fluent-plugin-concat
gem install fluent-plugin-detect-exceptions
gem install fluent-plugin-detect-exceptions-with-error
gem install fluent-plugin-record-modifier

And I am seeing the list of gems installed here:

opsmxuser@opsmx-ind-setup:~/docker-compose-efk$ fluent-gem list

*** LOCAL GEMS ***
.
.
fluent-plugin-concat (2.4.0)
fluent-plugin-detect-exceptions (0.0.12)
fluent-plugin-detect-exceptions-with-error (0.0.3)
fluent-plugin-record-modifier (2.0.1)
fluentd (1.6.2, 0.14.25)
.
.

But these 3 plugins are not enabled. When we are using these configurations into fluent.conf file getting following error messages into fluentd container logs.

2019-08-05 12:34:55 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf" 2019-08-05 12:34:55 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="Unknown filter plugin 'concat'. Run 'gem search -rd fluent-plugin' to find plugins"

2019-08-02 12:46:23 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'detect_exceptions'. Run 'gem search -rd fluent-plugin' to find plugins"

2019-08-05 12:34:55 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf" 2019-08-05 12:34:55 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown filter plugin 'record_modifier'. Run 'gem search -rd fluent-plugin' to find plugins"

uname0a uname0a · Accepted Answer · 2020-12-09T14:17:39

Add to fluentd config @type copy directive.
Example below:

    <match **>
  @type copy
  <store>
    @type exec
    @id load2clickhouse
    command bash /usr/local/bin/insert_ch.sh {{ template "fluentdInsertTable" $ }}
    format json
    <buffer>
      @type memory
      chunk_limit_size 32m
      queue_limit_length 32
      flush_at_shutdown true
      flush_interval 15s
      flush_thread_count 4
    </buffer>
  </store>
  <store>
    @type elasticsearch
    host ip
    port 5044
    logstash_format true
</store>
</match>

How to enable fluentd pluggins

1 Answers