How to parse a log string in Logstash using Grok?

Question

I am trying to parse the following string using Grok;

2018-06-08 13:26:02.002851: <action cmd="run" options="IGNORE_ERROR" path="/usr/lib/vmware/likewise/bin/lw-lsa get-metrics"> (/etc/vmware/vm-support/ad.mfx) took 0.000 sec

I want to separate the above out into columns ultimately like TIMESTAMP, ACTION, OPTIONS, PATH etc - I have tried multiple combinations but have so far failed.

You should be able to try and finally solve it using the grok debugger here: grokdebug.herokuapp.com — asatsi

Bhavya Jain Bhavya Jain · Accepted Answer · 2018-07-08T07:04:47

Grok pattern for above log:->

%{TIMESTAMP_ISO8601:time}:%{SPACE}\<%{WORD:action}%{SPACE} %{DATA:kvpairs}\>%{SPACE}\(%{DATA:path_2}\)%{SPACE}took%{SPACE}%{NUMBER:time_taken}%{SPACE}%{WORD:time_unit}

In the above grok pattern, I have captured cmd, options and path in an event named kvpairs. This is because these key-value pairs can be easily extracted in logstash using kv filter. So your filter configuration will look like:->

filter{
    grok(
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}:%{SPACE}\<%{WORD:action}%{SPACE} %{DATA:kvpairs}\>%{SPACE}\(%{DATA:path_2}\)%{SPACE}took%{SPACE}%{NUMBER:time_taken}%{SPACE}%{WORD:time_unit}"}
    )

    kv{
        source => "kvpairs"
    }

    date{
        match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSS"]
    }
}

kv filter by default takes space as the delimiter and will extract columns cmd,options and path. date filter will make the @timestamp variable.

How to parse a log string in Logstash using Grok?

1 Answers