Making strings not_analyzed for Kibana

Question

I want my strings to be not_analyzed both for search and for visualizations in Kibana.

I created a custom elasticsearch-template.json to set the default for strings to be not_analyzed, and I pointed to this in my logstash-log4j.conf file. Here is the elasticsearch-template.json:

{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "not_analyzed", "omit_norms" : true,
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}

I have this file in the same directory as my logstash-log4j.com which contains:

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    cluster => 'hlt_logs'
    index => 'logstash-symphony'
    template => "/elk/logstash/current/elasticsearch-template.json"
  }
}

I removed and replaced the elasticsearch index using curl after making these changes:

   curl -XDELETE -i 'http://localhost:9200/logstash-symphony'
   curl -XPUT -i 'http://localhost:9200/logstash-symphony'

I added some new logs and went to visualize in Kibana, but I'm still getting my strings as analyzed fields. What am I missing?

So ES's get mapping API reports that the mappings of the recreated index is wrong? Have you verified with the get index template API that Logstash has indeed updated the index template before you recreated the index? — Magnus Bäck
check out the mapping with: curl -XGET 'localhost:9200/logstash-symphony/_mapping/?pretty=true' — markus
@markus under what circumstances will that need to be re-done? If we stop and restart kibana? If we reboot the host machine? — PurpleVermont
@MagnusBäck I'm not sure how to do that. Can you point me to the relevant bits of the documentation? — PurpleVermont
@PurpleVermont Can you see the correct mapping in your index with the command markus put? If the mapping is correct, did you try to refresh the index in Kibana to get the new configuration? — Pigueiras

bd808 bd808 · Accepted Answer · 2015-11-06T05:27:03

The template parameter of the elasticsearch output plugin can be used to provide your index template to be used by Elasticsearch when creating a new Logstash index.

Making strings not_analyzed for Kibana

1 Answers