Elasticsearch Index template lost raw string mapping

Question

I'm running a small ELK 5.4.0 stack server on a single node. When I started, I just took all the defaults, which meant 5 shards for each index. I didn't want the overhead of all those shards, so I created an index template like so:

PUT /_template/logstash
{
  "template": "logstash*",
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}

This worked fine, but I just realized that all my raw fields are now missing in ES. For example, "uri" is one of my indexed fields and I used to get "uri.raw" as an unanalyzed version of it. But since I updated the template, they are missing. Looking at the current template shows

GET /_template/logstash
Returns:
{
    "logstash": {
        "order": 0,
        "template": "logstash*",
        "settings": {
            "index": {
                "number_of_shards": "1",
                "number_of_replicas": "0"
            }
        },
        "mappings": {},
        "aliases": {}
    }
}

It seems that the mappings have gone missing. I can pull the mappings off an earlier index

GET /logstash-2017.03.01

and compare it with a recent one

GET /logstash-2017.08.01

Here I see that back in March there was a mapping structure like

mappings: {
    "logs": {
        "_all": {...},
        "dynamic_templates": {...},
        "properties": {...}
    },
    "_default_": {
        "_all": {...},
        "dynamic_templates": {...},
        "properties": {...}
    }
}

and now I have only

mappings: {
    "logs": {
        "properties": {...}
    }
}

The dynamic_templates hash holds the information about creating "raw" fields.

My guess is that I need to add to update my index template to

PUT /_template/logstash
{
  "template": "logstash*",
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "logs": {
        "_all": {...},
        "dynamic_templates": {...},
    },
    "_default_": {
        "_all": {...},
        "dynamic_templates": {...},
        "properties": {...}
    }
}

IOW, everything but logs.properties (which holds the current list of fields being sent over by logstash).

But I'm not an ES expert and now I'm a bit worried. My original index template didn't work out the way I thought it would. Is my above plan going to work? Or am I going to make things worse? Must you always include everything when you create an index template? And where did the mappings for the older indexes, before I had a template file, come from?

Val Val · Accepted Answer · 2017-08-04T03:59:03

When Logstash first starts, the elasticsearch output plugin installs its own index template with the _default_ template and dynamic_templates as you correctly figured out.

Everytime Logstash creates a new logstash-* index (i.e. every day), the template is leveraged and the index is created with the proper mapping(s) present in the template.

What you need to do now is simply to take the official logstash template that you have overridden and reinstall it like this (but with the modified shard settings):

PUT /_template/logstash
{
  "template" : "logstash-*",
  "version" : 50001,
  "settings" : {
    "index.refresh_interval" : "5s"
    "index.number_of_shards": 1,
    "index.number_of_replicas": 0
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "norms" : false},
      "dynamic_templates" : [ {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text",
            "norms" : false
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text", "norms" : false,
            "fields" : {
              "keyword" : { "type": "keyword", "ignore_above": 256 }
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date", "include_in_all": false },
        "@version": { "type": "keyword", "include_in_all": false },
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }
      }
    }
  }
}

Another way you could have done it is to not overwrite the logstash template, but use any other id, such as _template/my_logstash, so that at index creation time, both templates would have kicked in and used the mappings from the official logstash template and the shard settings from your template.

Elasticsearch Index template lost raw string mapping

1 Answers