How to parse the custom logs

Question

I am new to logstash , can someone help me on grok filter to parse the data from multiple newline characters in the same log

2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion() url: GET::/system/data/connect/service response: 200 elapsed: 10 ms

2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion() \n response: 200 \n elapsed: 10 ms — jag m

IRJSK IRJSK · Accepted Answer · 2018-10-09T05:38:36

1.Using Grok

http://grokdebug.herokuapp.com/

[First Input Box] INPUT

2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()
response: 200
elapsed: 10 ms

[Second Input Box] Grok Parse ==>%{UPTONEWLINE:Part1}%{UPTONEWLINE:Part2}

Check Add custom patterns and add the following line UPTONEWLINE (?:(.+?)(\n))

OUTPUT

{
  "Part1": [
    [
      "2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()\n"
    ]
  ],
  "Part2": [
    [
      "response: 200\n"
    ]
  ]
}

2.Without using Grok filter - Logstash configuration file

INPUT

2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()\nresponse: 200\nelapsed: 10 ms

Logstash Config File

input {
  http {
    port => 5043
    response_headers => {
        "Access-Control-Allow-Origin" => "*"
        "Content-Type" => "text/plain"
        "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, 
        Accept"
    }   
    }
}

filter {
    mutate {           
            split => ['message','\n']
            add_field => {
                 "Part1" => "%{[message][0]}"
                 "Part2" => "%{[message][1]}"
                 "Part3" => "%{[message][2]}"
            }
         }
}

output {
  stdout  { 
        codec => rubydebug
    }
}

OUTPUT

{
  "host"=>"0:0:0:0:0:0:0:1",
  "@version"=>"1",
  "message"=>[
    [0]"2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exe c-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()",
    [1]"response: 200",
    [2]"elapsed: 10 ms"
  ],
  "Part1"=>"2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()",
  "Part2"=>"response: 200",
  "Part3"=>"elapsed: 10 ms",
  "@timestamp"=>2018-10-09T05: 27: 41.695Z
}

How to parse the custom logs

1 Answers