Fluentd Elasticsearch target index

6
votes

I'm using Fluentd to transfer the data into Elasticsearch.

td-agent.conf

## ElasticSearch
<match es.**>
  type elasticsearch
  target_index_key @target_index  
  logstash_format true
  flush_interval 5s
</match>

Elasticsearch index : 

"logstash-2016.02.24" : {
    "aliases" : { },
    "mappings" : {
      "fluentd" : {
        "dynamic" : "strict",
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "dummy" : {
            "type" : "string"
          }

        }
      }
    },

Transmit json data :

$ curl -X POST -d 'json={"@target_index": "logstash-2016.02.24","dummy":"test"}' http://localhost:8888/es.test

It should write the data to the given index instead of that It creates new index - logstash-2016.02.25 and it will write data into that. I want to write data to the given index.

Here is the Fluentd elasticsearch github link : https://github.com/uken/fluent-plugin-elasticsearch

Please correct me if I'm missing something.

3
elasticsearchkibanafluentd
I think you have a typo, type elasticsearch should read @type elasticsearch. – Val
I'm using td-agent v2 on centOS, it's working properly without @. – Nimit

3 Answers

3
votes

try this, its due to logstash_format true, please enter your index name in below index_name field (default value is fluentd)

<match es.**>
@type elasticsearch
host localhost
port 9200
index_name <.....your_index_name_here.....>
type_name fluentd
flush_interval 5s
</match>

after run this, please check index created or not by load below url in your browser

http://localhost:9200/_plugin/head/

have a good luck

3
votes

Maybe this is old but actually I run into the same problem and solved by

logstash_format false
index_name fluentd

This create only fluentd as index. From official fluentd tutorial https://docs.fluentd.org/output/elasticsearch

logstash_format (optional): With this option set true, Fluentd uses the conventional index name format logstash-%Y.%m.%d (default: false). This option supersedes the index_name option.

For clean up of old indices please consider using Curator: https://github.com/elastic/curator

I hope it helps someone.

2
votes

Its because you set logstash_format true, so you have to set the logstash_prefix.

Apache httpd example:

  logstash_prefix fluentd.httpd # defaults to "logstash"
  logstash_prefix_separator _   # defaults to "-"