LogStach-Grok: How to match Apache TimeStamp

Question

I have no idea how to match my TimeStamp. Could someone please help me?

Example of my TimeStamps from Apache Log: "2016-06-13T14:54:39.000+0100"

filter {   if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }   }
    date {
      timezone     => "UTC"
      match        => [ "timestamp" , "yyyy-mm-ddTHH:mm:ss Z"]
    }   } }

output {   stdout { codec => rubydebug } }

Output:

{:timestamp=>"2016-06-13T14:56:43.196000+0100", :message=>"Error: Cannot register filter date plugin. The error reported is: \n Illegal pattern component: T for pattern 'yyyy-mm-dd\THH:mm:ss Z'", :level=>:error}

Apache Log Example:

{ "@version": "1", "@timestamp": "2016-06-14T09:11:23.000+0100", "message": "GET /page1/page2/ HTTP/1.1", "via": "192.168.1.1", "client-ip": "192.168.1.23", "remote-logname": "-", "remote-user": "-", "recv-time": "[14/Jun/2016:09:11:23 +0100]", "serve-time-microsec": "85471", "request": "GET /page1/page2/ HTTP/1.1", "status": "200", "size": "79648", "referer": "http://www.google.com/", "user-agent": "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13", "url": "/page1/page2/", "query": "", "method": "GET", "protocol": "HTTP/1.1", "vhost": "www.site.com", "received-size": "998" }

Can you add an example of line from your log? it will be easier to check. — ItayD
There is already a line from my log @ItayD. I am having an issue with matching this date format 2016-06-13T14:54:39.000+0100 — Narayana
I don't see any line, and i'm asking it because maybe you can use a better grok matcher than COMBINEDAPACHELOG which uses date format %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}. — ItayD
Anyway, the format of your original timestamp is: yyyy-MM-dd'T'HH:mm:ss.SSSZ — ItayD
Thank you that worked @ItayD (The error message has disappeared)! I have added an example of a full Apache Log. I see that the TimeStamp in my Kibana interface is not changing though. Could I ask you why that would be ? — Narayana

ItayD ItayD · Accepted Answer · 2016-06-14T08:38:23

From your full log I can see that its already in Json format, so you don't need to use grok filter on message field using COMBINEDAPACHELOG pattern. Just use the date filter with the following format:

date {
     timezone  => "UTC"
     match     => [ "timestamp" , "yyyy-mm-dd'T'HH:mm:ss.SSSZ"]
}

This is my full logstash.conf file which worked with your example:

input {
    file {
      path => "/var/log/test.log"
      codec => json
      start_position => "beginning"
    }
}
filter {
    date {
          timezone     => "UTC"
          match        => [ "timestamp" , "yyyy-mm-dd'T'HH:mm:ss.SSSZ"]
        }
}
output {
    stdout { 
        codec => rubydebug 
    }
}

LogStach-Grok: How to match Apache TimeStamp

1 Answers