filebeat-index-template.json for ElasticSearch 6.2.4

Question

I am running ElasticSearch 6.2.4. I tried to create Filebeat index template, but got the following error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "No handler for type [string] declared on field [message]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_default_]: No handler for type [string] declared on field [message]",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "No handler for type [string] declared on field [message]"
    }
  },
  "status" : 400
}

filebeat-index.template.json

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "not_analyzed",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "string",
          "index": "analyzed"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        },
        "geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "location" : { "type" : "geo_point" }
          }
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

I wonder if there is official filebeat-index-template.json that work for ElasticSearch 6.2.4

Other thing that I have tried

Try filebeat -c "/etc/filebeat/filebeat.yml" export template > filebeat.template.json, but the filebeat will keep running forever without creating anything.
I've tried to change "type": "string" to "type": "text",, but got another error where _all is deprecated.
I've also tried to remove _all, but ElasticSearch keep have parsing error when Logstash send data to ElasticSearch.

Filebeat Version [Old]

I also try to find out the version of my Filebeat. I tried

> filebeat -v
Loading config file error: Failed to read /root/filebeat.yml: open /root/filebeat.yml: no such file or directory. Exiting.

> filebeat -v -c "/etc/filebeat/filebeat.yml"
(it struck forever)

I am following this https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04, but instead of using ElasticSearch 2.0 and Kibana 4.5, I am installing ElasticSearch 6.2.4, Kibana 6.2.4, and Logstash 6.2.4 and Ubuntu 16.04.4 LTS

Upgrading to Filebeat 6.2.4

Now I am upgrading Filebeat to 6.2.4. Now I get this error

Exiting: Could not start registrar: Error loading state: Error decoding states: json: cannot unmarshal object into Go value of type []file.State

I removed this error by rm /var/lib/filebeat/registry. Now I can do filebeat export template > template.json and it work fine now. I will close the question soon.

@Val, I have added the answer to your question in the updated post. — invisal

simples simples · Accepted Answer · 2018-08-07T22:56:14

Try to use this elastic 6.0 modified json for filebeat-index.template.json

{
  "mappings": {
    "_default_": {
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "false",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text",
          "index": "true"
        },
        "offset": {
          "type": "long",
          "doc_values": "true"
        },
        "geoip": {
          "type": "object",
          "dynamic": true,
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

Basically I changed the message type from string to text. Also from elastic 6.0 onwards the index field uses true or false, instead of analyzed.

After running this command (as suggested in the blog you are referring to above):

curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' [email protected] -H 'Content-Type: application/json'

I managed to get the correct confirmation from elastic:

{ 
  "acknowledged" : true
}

I haven't tested it yet, but please let me know if it works for you.

You will probably notice that the _all template is also removed from the original json. Why? Apparently it was depreciated in elastic 6.0 and there are ways to use copy_to instead as suggested in here but I haven't figured it out yet.

filebeat-index-template.json for ElasticSearch 6.2.4

Other thing that I have tried

Filebeat Version [Old]

Upgrading to Filebeat 6.2.4

2 Answers