Add raw fields in elasticsearch from fluentd

Question

I have setup a centralized logging system where my application nodes have FluentD installed, tailing log files. FluetndD on the application nodes forward the processed logs to FluentD on a collector node. FluentD on this collector node then forwards to a variety of outputs, one being an ElasticSearch cluster. I then run Kibana to read from the ES. (Fairly common setup I believe.)

When trying to visualize some of the data in Kibana on certain data fields, Kibana is not grouping things correctly. For example, an email address is considered multiple values (name, domain, etc.). After doing some research it is my understanding that the problem is that my fields need to be configured to be replicated as .raw values as well. (Analyzed versus non-analyzed)

The problem is I cannot figure out how to get FluentD to add this data into Elasticsearch to include the .raw fields. Supposedly this is something Logstash does by default?

I am using the fluent-plugin-elasticsearch: https://github.com/uken/fluent-plugin-elasticsearch

fylie fylie · Accepted Answer · 2016-09-24T15:34:16

To stop Elasticsearch from tokenizing your field, you can specify the field as not_analyzed by updating the index mapping. You'll want to delete the index (or create a new one) to see this change take effect, since you can't modify an existing mapping in an existing index. You'll want to put a mapping that contains the following:

{
  "email": {
    "type":     "string",
    "index":    "not_analyzed"
  }
}

Add raw fields in elasticsearch from fluentd

1 Answers