convert nginx logs to elasticsearch using grock in logstash

Question

I have the following message from nginx:

    XXX.XXX.XXX.XXX - - [09/Jul/2014:15:23:51 +0200] "GET /js/order.js HTTP/1.1" 200 2777 "http://www.yyy.xxx.zz/accueil"; "Mozilla/5.0 (Windows NT 5.1; rv:30.0) Gecko/20100101 Firefox/30.0" "-" "0.000" "317"

I need to collect the data to elasticsearch document:

httpversion 1.1
method      GET
offset      127904969
request     /js/order.js
response    200
timestamp   09/Jul/2014:15:23:51 +0200
type        nginx
url         http://www.geoportail.gouv.fr/accueil

In order to achieve that, I am using the following pattern:

match => [ "message", "%{IPORHOST:clientip} - - \[%{log-date:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER\:response} (?:%{NUMBER:bytes}|-) \"%{URI:url}" ]

The problem I have is that all the fields are seen as string:

"method"   :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"offset"   :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"request"  :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"response" :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"timestamp":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"type"     :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"url"      :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}}

I would like to set the timestamp field as "date" or even better, to replace the @timestamp with this timestamp in order to be able to query it with the range api of elasticsearch for example.

Following the advice of Alcanzar, I modified the conf file as follow:

input {
  lumberjack {
    # The port to listen on                                                                                                                                  
    port => 5140

    # The paths to your ssl cert and key                                                                                                                     
    ssl_certificate => "/etc/logstash/logstash.crt"
    ssl_key => "/etc/logstash/logstash.key"

    # Set this to whatever you want.                                                                                                                         
    type => "nginx"
  }
}

filter {
  grok {
    match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER\
:response} (?:%{NUMBER:bytes}|-) \"%{URI:url}" ]
  }
}

filter {
 date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] }
}

output {
  elasticsearch {
    host => localhost
    index => front_gpp3
  }
  stdout { codec => rubydebug }
}

The problem was the locale, my VM had FR as locale instead of EN, so I add to add this to the filter:

filter {
  grok {
    match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER\
:response} (?:%{NUMBER:bytes}|-) \"%{URI:url}" ]
  }
  date {
    locale => "en"
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

Alcanzar Alcanzar · Accepted Answer · 2014-07-11T17:22:40

you need to add a date filter:

filter {
 date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] }
}

or something close to that. That will parse timestamp using that format and then set @timestamp using the parsed value.

convert nginx logs to elasticsearch using grock in logstash

1 Answers