Inserting a dummy value for empty field while parsing using grok logstash

1
votes

i am trying to parse logs and put it inside elastic search using logstash.

my log file is on the below format

[18-Aug-2016 02:28:46,537][ERROR][thread1][package.name] there is error in line 52

\[%{GREEDYDATA:date} %{GREEDYDATA:time}\]\[%{LOGLEVEL:log_type}\]\[%{GREEDYDATA:thread_name}\]\[%{GREEDYDATA:package}\](%{GREEDYDATA:log_msg})?

when i run this grok filter i get the output properly. However , there are instances where i get the input without last field (log_msg). something like this:

[18-Aug-2016 02:28:46,537][ERROR][thread1][package.name]

In this case grok is ignoring the last field log_msg and that is not getting inseted into elastic search.

But, is there any way , we can set an empty string or string saying "no data" for log_msg field if that doesn't exist in the message.

Real Ouput:

{
        "message" => "[18-Aug-2016 02:28:46,537][ERROR][thread1][package.name]",
       "@version" => "1",
     "@timestamp" => "2016-08-17T12:31:58.209Z",
           "path" => "/home/admin-nfv/test1_log.log",
           "host" => "nendc1-bg-d104",
           "date" => "18-Aug-2016",
           "time" => "02:28:46,537",
       "log_type" => "ERROR",
    "thread_name" => "thread1",
        "package" => "package.name"
}

Expected Output:

{
        "message" => "[18-Aug-2016 02:28:46,537][ERROR][thread1][package.name]",
       "@version" => "1",
     "@timestamp" => "2016-08-17T12:31:58.209Z",
           "path" => "/home/admin-nfv/test1_log.log",
           "host" => "nendc1-bg-d104",
           "date" => "18-Aug-2016",
           "time" => "02:28:46,537",
       "log_type" => "ERROR",
    "thread_name" => "thread1",
        "package" => "package.name",
        "log_msg" => "no data"
}
1
elasticsearchlogstashlogstash-grok

1 Answers

4
votes

You can add a mutate filter that will add an empty field if it is not present:

filter {
    if ![log_msg] {
        mutate {
            add_field => {"log_msg" => "no data" }
        }
    }
}