Fluentd on kubernetes : Segregating container logs based on container name and retag them to send to ElasticSearch

Question

I have deployed fluentd in Openshift cluster and setup ES and Kibana On-Premise. I need to capture logs from the nodes and transmit them to ES running on-prem. Specifically, I need to separate the /var/log/containers/*.log into two sections based on a container name (kong)so that all kong logs are tagged as kong and remaining ones are tagged as application. Additionally, I would also require the kubernetes metadata info for the pod logs(namespace, container name etc)

Is there a way to achieve that?

Fluentd docker image version: fluent/fluentd-kubernetes-daemonset:v1.11.1-debian-elasticsearch7-1.3 ElasticSearch, Kibana - 7.8.0

Below are my configuration files. fluentd.conf

    # AUTOMATICALLY GENERATED
    # DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb

    @include kubernetes.conf

    <match **>
        @type elasticsearch
        @log_level info
        include_tag_key true
        host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
        port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
        scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
        logstash_format true
        logstash_prefix openshift-${tag}
        user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
        password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"
        <buffer>
         @type file
         path /var/log/fluentd-buffers/kubernetes.system.buffer
         retry_max_interval 30
         flush_interval 1s
         flush_thread_count 8
         chunk_limit_size 2M
         queue_limit_length 32
         overflow_action block
         retry_forever true
        </buffer>
    </match>

kubernetes.conf

    <label @FLUENT_LOG>
      <match fluent.**>
        @type null
      </match>
    </label>

    <source>
      @type tail
      path /var/log/containers/*kong*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag "#{ENV['FLUENT_CONTAINER_TAIL_TAG'] || 'kubernetes.*'}"
      exclude_path ["/var/log/containers/fluentd*"]
      read_from_head true
      <parse>
      @type multi_format
        <pattern>
          format json
          time_format '%Y-%m-%dT%H:%M:%S.%N%Z'
          keep_time_key true
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%N%:z'
          keep_time_key true
        </pattern>
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
      kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
      verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
      ca_file "#{ENV['KUBERNETES_CA_FILE']}"
      skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
      skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
      skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
      skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
    </filter>

I tried changing the tag name kubernetes.* to kong . I was able to get the log in ES but the kubernetes metadata were missing.

Please help in this regard.

Missed to add an info that the kong logs and non-kong logs need to be sent to separate ES instances. — karthik ravi
Got this working at last. Created more filters and made it work! — karthik ravi

psicopante psicopante · Accepted Answer · 2020-09-05T19:19:06

my solution was to set a label. See this, I guarantee it will help you.

https://stackoverflow.com/a/63739163/3740244

\o/Good is that, by boyBR

Fluentd on kubernetes : Segregating container logs based on container name and retag them to send to ElasticSearch

1 Answers