geoip.location is defined as an object in mapping [doc] but this name is already used for a field in other types

Question

I'm getting this error:

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"nginx-access-2018-06-15", :_type=>"doc", :_routing=>nil}, #], :response=>{"index"=>{"_index"=>"nginx-access-2018-06-15", "_type"=>"doc", "_id"=>"jo-rfGQBDK_ao1ZhmI8B", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[geoip.location] is defined as an object in mapping [doc] but this name is already used for a field in other types"}}}}

I'm getting the above error but don't understand why, this is loading into a brand new ES instance with no data. This is the first record that is inserted. Why am I getting this error? Here is the config:

input {

  file {
    type => "nginx-access"
    start_position => "beginning"
    path => [ "/var/log/nginx-archived/access.log.small"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  if [type] == "nginx-access" {
       grok {
         patterns_dir => "/etc/logstash/patterns"
         match => { "message" => "%{NGINX_ACCESS}" }
         remove_tag => ["_grokparsefailure"]
       }
       geoip {
         source => "visitor_ip"
       }
       date {
         # 11/Jun/2018:06:23:45 +0000
         match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
         target => "@request_time"
       }
       if "_grokparsefailure" not in [tags] {
            ruby {
              code => "
                thetime = event.get('@request_time').time
                event.set('index_date', 'nginx-access-' + thetime.strftime('%Y-%m-%d'))
              "
            }
          }
         if "_grokparsefailure" in [tags] {
              ruby {
                code => "
                  event.set('index_date', 'nginx-access-error')
                "
              }
            }
     }

}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "%{index_date}"
    template => "/etc/logstash/templates/nginx-access.json"
    template_overwrite => true
    manage_template => true
    template_name => "nginx-access"
  }
  stdout { }
}

Here's a sample record:

{
            "method" => "GET",
          "@version" => "1",
             "geoip" => {
         "continent_code" => "AS",
               "latitude" => 39.9289,
           "country_name" => "China",
                     "ip" => "220.181.108.103",
               "location" => {
             "lon" => 116.3883,
             "lat" => 39.9289
         },
            "region_code" => "11",
            "region_name" => "Beijing",
              "longitude" => 116.3883,
               "timezone" => "Asia/Shanghai",
              "city_name" => "Beijing",
          "country_code2" => "CN",
          "country_code3" => "CN"
     },
        "index_date" => "nginx-access-2018-06-15",
            "ignore" => "\"-\"",
             "bytes" => "2723",
           "request" => "/wp-login.php",
     "@request_time" => 2018-06-15T06:29:40.000Z,
           "message" => "220.181.108.103 - - [15/Jun/2018:06:29:40 +0000] \"GET /wp-login.php HTTP/1.1\" 200 2723 \"-\" \"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
              "path" => "/var/log/nginx-archived/access.log.small",
        "@timestamp" => 2018-07-09T01:32:56.952Z,
              "host" => "ab1526efddec",
        "visitor_ip" => "220.181.108.103",
         "timestamp" => "15/Jun/2018:06:29:40 +0000",
          "response" => "200",
          "referrer" => "\"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
       "httpversion" => "1.1",
              "type" => "nginx-access"
 }

mikeb mikeb · Accepted Answer · 2018-07-10T11:57:53

Figured out the answer, based on this:

https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html#_schedule_for_removal_of_mapping_types

The basic problem is that for each Elasticsearch index, each field must be the same type, even if the records are different types.

That is, if I have a person { "status": "A" } stored as text I cannot have a record for a car { "status": 23 } stored as a number in the same index. Based on the info in the link above, I'm storing one "type" per index.

My output section for Logstash looks like this:

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "%{index_date}"
    # Can test loading this with:
    # curl -XPUT -H 'Content-Type: application/json' -d@/docker-elk/logstash/templates/nginx-access.json http://localhost:9200/_template/nginx-access
    template => "/etc/logstash/templates/nginx-access.json"
    template_overwrite => true
    manage_template => true
    template_name => "nginx-access"
  }
  stdout { }
}

My template looks like this:

{
  "index_patterns": ["nginx-access*"],
  "settings": {

  },
  "mappings": {
    "doc": {
      "_source": {
        "enabled": true
      },
      "properties": {
    "type" : { "type": "keyword" },
    "response_time": { "type": "float" },
        "geoip" : {
      "properties" : {
        "location": {
              "type": "geo_point"
            }
          }
    }
      }
    }
  }
}

I'm also using the one type per index pattern described in the link above.

geoip.location is defined as an object in mapping [doc] but this name is already used for a field in other types

1 Answers