Elasticsearch 5.x has no mechanism to silently drop fields that don't conform to particular type based (given mapping)?

Question

Elasticsearch docs ( https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html) state the following:

By default, when a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping.

So, if we create a document (for which no index/type mappings exist before hand) like this:

curl -X POST 'http://localhost:9200/my_index/food/1' -d \
'{
  "name": "pie",
  "delicious": true,
  "age": 100.5
}'

the types are discovered automatically and the mappings for the type food in index my_index become:

{
  "my_index": {
    "mappings": {
      "food": {
        "properties": {
          "age": { "type": "float" },
          "delicious": { "type": "boolean" },
          "name": {
            "type": "text",
            "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
          }
        }
      }
    }
  }
}

If I attempt to add a new document to index/type my_index/food for which one of the field values violates the 'contract' of the mapping specification then I find elasticsearch returns an error and declines to index the offending document.

curl -X POST 'http://localhost:9200/my_index/food/2' -d \
'{
  "name": "goat",
  "delicious": false,
  "age": true
}'

leads to:

 mapper_parsing_exception","reason":"failed to parse [age]"}],"type":"mapper_parsing_exception","reason":"failed to parse [age]","caused_by":{"type":"json_parse_exception","reason":"Current token (VALUE_TRUE) not numeric, can not use numeric value accessors\n

My question is: is there any way to configure elastic search so that my attempt to index this document: { "name": "goat", "delicious": false, "age": true }

would just drop the (improperly typed) field 'age', and index the rest of the given document as follows:

'{ "name": "goat", "delicious": false}'

I'm guessing no search feature is available, but wanted to check. Thanks in advance!

John Jones John Jones · Accepted Answer · 2017-04-05T19:58:48

I'm pretty new to ElasticSearch and doubly so for the +5.x branch, but I just stumbled on the ignore_malformed mapping parameter after reading your question.

Sometimes you don’t have much control over the data that you receive. One user may send a login field that is a date, and another sends a login field that is an email address.

Trying to index the wrong datatype into a field throws an exception by default, and rejects the whole document. The ignore_malformed parameter, if set to true, allows the exception to be ignored. The malformed field is not indexed, but other fields in the document are processed normally.

This last detail seems to describe your use case, not sure, been huffing too much solder fumes this week. :)

Here's the docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html

Elasticsearch 5.x has no mechanism to silently drop fields that don't conform to particular type based (given mapping)?

1 Answers