Timestamp not getting my log value using date filter logstash?

0
votes

I have following config to fetch Http call and log date at time call was made, but @timestamp shows when log was send from FileBeat server to logstash one.

 input {
        beats {
                port => 5044
        }
}

filter {
         if [message] !~ /Calling SMC REST API/ {
                 drop { }
        }else {
                grok {
                        match => ["message", "{URIPATHPARAM:request}"]
                        match => ["message", "\[%{DATA:timestamp}\]%{GREEDYDATA:extra} %{GREEDYDATA:date}"] 
                }

                date {
                        match => ["timestamp", "[YYYY-MM-dd HH:mm:ss,SSS]"]
                        target => "timestamp"
                }

        }


}

output {
        elasticsearch {
                hosts => ["http://localhost:9200/"]
                index => "smc_calls-%{+YYYY.MM.dd}"

        }

stdout { codec => rubydebug }
}

Input log message:

[2017-04-25 12:41:25,064] rf234234232345454:Sg2343 INFO emcJmsContainer-2 - ******Calling SMC REST API******** http://com.asds.dsds:45/api?search=query

Output in ruby:

"date" => "http-call", --> dont why http call is shown here
        "offset" => 18036878,
    "input_type" => "log",
        "source" => "log source",
       "message" => "entire filtered message",
          "type" => "log",
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_dateparsefailure"
    ],
    "@timestamp" => 2017-05-03T06:08:59.486Z,
         "extra" => "reaminaing data from message",
      "@version" => "1",
          "beat" => {
        "hostname" => "host",
            "name" => "host",
         "version" => "5.3.1"
    },
          "host" => "icgaspadsw01d",
     "timestamp" => "2017-04-25 12:41:25,064"  --> log timestamp i want but dont see in kibana when indexed
2
logstashlogstash-grok
Your grok parser is written incorrectly so the field timestamp will not contain a date, it will probably contain nothing or the entire log message. Try using a tool like grok constructor to get a working parser and then you will probably be able to solve this one yourself. grokconstructor.appspot.com/do/match - Will Barnwell

2 Answers

1
votes

I think your grok parser is written in wrong manner. Use this in grok 

\[%{DATA:timestamp}\]%{GREEDYDATA:extra} %{GREEDYDATA:date}

and then use your date filter.

0
votes

Got that working with following grok plus date filter:

grok {
                        match => ["message", "{URIPATHPARAM:request}"]
                        match => ["message", "%{TIMESTAMP_ISO8601:time}"]
                }

                 date {
                        match => ["time", "YYYY-MM-dd HH:mm:ss,SSS"]
                }