Logstash "date" filter doesn't replace @timestamp

Question

For some reason when I try to replace the @timestamp field in my apache access log events, it doesn't work. For Nginx it did work with the same date format.

This is my log date format : [10/Dec/2014:13:01:06 -0500]

This is my logstash.conf -

if [type] == "apache" {
   grok {
        match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }
        add_tag => [ "apache", "grokked" ]
        }
   date {
        match => [ "[time]" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        locale => "en"
        add_tag => ["date_filtered"]
        }
     }

The @timestamp field stays in this format - 2014-12-10T18:01:06.000Z

update: This is the full configuration -

There are no errors in the logs, only logstash startup log lines.

My logstash configuration file is as follows :

input {
redis {
        host => "<redis_server>"
        port => "<redis_port>"
        data_type => 'list'
        key => 'logstash'
        type => 'redis'
        threads => 300
 }

filter {


multiline {
        pattern => "(^\s)|(^INFO.+)|(^SEVERE.+)"
        what => "previous"
        add_tag => ["multiline"]
        stream_identity => '%{host}.%{file}.%{type}'

   }

if [type] == "apache" {
   grok {
        match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:time}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }
        add_tag => [ "apache", "grokked" ]
        }
   date {
        match => [ "[time]" , "dd/MMM/YYYY:HH:mm:ss Z" ]
        locale => "en"
        add_tag => ["date_filtered"]
        }
     }
}

output {

  elasticsearch {
    host => "<ES_server>"
    port => "9300"
    cluster => "elasticsearch"
    embedded => false
    node_name => "logstash"
   }
}

And the result is :

enter image description here

When it should actually be :

enter image description here

Thanks.

I don't see time in your grok filter ? how are you extracting time out of your message ? — MUFC
@Vaibhav: \[%{HTTPDATE:time}\] is included in the pattern. — Magnus Bäck
@user3323914: Not sure exactly what you're asking here. Your last statement indicates that you're expecting the timestamp field to have another format, but the format of that field is always ISO8601 as in your example. Or is the problem that the date filter fails to parse the timestamps? If so the log will contain an error message that points in the right direction. — Magnus Bäck
@MagnusBäck, Thanks. If I understand you correctly, these are 2 different formats? - [10/Dec/2014:13:01:06 -0500] and 2014-12-10T18:01:06.000Z ? Do you know what should I do to make it work with the current format? Thanks! — deez

Alain Collins Alain Collins · Accepted Answer · 2014-12-14T07:33:24

On average, only 1/1000 logs are processed at millisecond=000 as shown in your @timestamp.

Can you guess what happens when you pass HH:mm:ss into date{}? That's right - with no millisecond information provided in the input, it's set to 000.

So, I think your date{} stanza is working just fine.

You can confirm this by looking for the tag "date_filtered" in your output; the tag will only be added if the date{} succeeds.

Logstash "date" filter doesn't replace @timestamp

1 Answers