Elasticsearch does not understand Logstash types

Question

I am using Logstash to output logs from a Tomcat access log into elasticsearch. The pattern I am using is as follows:

grok {
    type => "access_log"
    pattern  => "\[%{DATA:my_timestamp}\] %{IP:client} %{WORD:method} %{URIPATHPARAM:request} \[%{DATA:auth_data}\] \[%{DATA:another_timstamp}\] %{NUMBER:result_code} %{NUMBER:duration} %{NUMBER:bytes}"
  }

As you seen above the field duration us formatted as NUMBER.

When logging the information in elasticsearch and doing a REST query through Chrome sense plugin as follows:

GET /_all/_mapping?pretty=1

It identifies the field but with type String as follows:

 "duration": {
               "type": "multi_field",
               "fields": {
                  "duration": {
                     "type": "***string***",
                     "omit_norms": true
                  },
                  "raw": {
                     "type": "string",
                     "index": "not_analyzed",
                     "omit_norms": true,
                     "index_options": "docs",
                     "include_in_all": false,
                     "ignore_above": 256
                  }
               }
            }

I would expect it to understand it as a number. Is this expected?

For example other fields like geoip are understood as a different type than string.

 "geoip": {
               "dynamic": "true",
               "properties": {
                  "location": {
                     "type": "geo_point"
                  }
               }
            },

Thx in advance

Jettro Coenradie Jettro Coenradie · Accepted Answer · 2014-02-23T23:09:26

Hi I am almost certain log stash sends everything as a string. So if the value of the field is provided as a string (meaning surrounded by ".", than elasticsearch will make it a string. By default logstash creates a template mapping that makes raw fields for all string types. But it does not say anything about numbers. So if you want to to be a number, create your own custom mapping with an order 2 and add for the specific field duration configuration to be a number type:"long".

The default logstash mapping can be found here: https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

Creating your custom mapping is not hard, you can find an example here: http://blog.trifork.com/2014/01/28/using-logstash-elasticsearch-and-kibana-to-monitor-your-video-card-a-tutorial/

Elasticsearch does not understand Logstash types

1 Answers