(Kibana, logstash) How can I visualize Apache log data based on log's date and not logstash import date?

0
votes

I was trying make a search within newly imported Apache logs. However, pretty fast I realised that all the data was actually assigned to import date and not the date when log data was generated.

Datetimes in logs files do not play any role in Kibana, why? While importing, I used grok filter: combinedapachelog and also verified that it matched my logs.

To be clear I have two timestamp fields in Kibana/Logstahs data: '@timestamp' field (date) and 'timestamp' field (str)

First one is the import date I presume and latter could be the date in the log but on the other hand log's do not store timestamps but [MM/DD/YYYY HH:MM:SS] datetime (or similar).

EDIT: I turned debug on and it showed _dateparsefailure or similar. I made few small shanges to filter and it went away- However, now that date is parsed fine the time is wrong and set 00:00:00.

1
apachedatetimelogginglogstashkibana

1 Answers

0
votes

It's been a long time since I worked on logstash. My current configuration that parses apache logs use the timestamp str field. 

if [type] == "apache-access" {
        grok {
                match => ["message", "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer}; %{QS:agent}",
                                                        "message", "%{COMBINEDAPACHELOG}"
                ]
                add_field => ["received_from", "%{[host]}"]
        }

        date {
                match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
                locale => "en"
                timezone => "Asia/Kolkata"
                add_tag => ["tsmatch"]
        }
}