Grok filter for application logs

Question

In my application I have log fromat as follows-

logFormat: '%-5level [%date{yyyy-MM-dd HH:mm:ss,SSS}] [%X{appReqId}] [%X{AppUserId}] %logger{15}: %m%n'

and the output of that format is like

INFO  [2017-02-03 11:09:21.792372] [b9c0d838-10b3-4495-9915-e64705f02176] [ffe00000000000003ebabeca] r.c.c.f.r.MimeTypeResolver: [Tika MimeType Detection]: filename: 'N/A',  detected mime-type: 'application/msword', time taken: 2 ms

Now I want each field of the log to be queryable at kibana and for that i want logstash to parse the input log message and it seems grok filter is there to help us.If grok filter is able to filter my message properly output should be like

"message" => "INFO  [2017-02-03 11:09:21.792372] [b9c0d838-10b3-4495-9915-e64705f02176] [ffe00000000000003ebabeca] r.c.c.f.r.MimeTypeResolver: [Tika MimeType Detection]: filename: 'N/A',  detected mime-type: 'application/msword', time taken: 2 ms",
"appReqId" => "b9c0d838-10b3-4495-9915-e64705f02176",
"timestamp" => "2017-02-03 11:09:21.792372",
"AppUserId" => "ffe00000000000003ebabeca",
"logger" => "r.c.c.f.r.MimeTypeResolver",

I am not able to figure it out how shall i configure at logstash.conf file so that i get the desired output. I tried like following

filter {
 grok {
match => { "message" => "%{LOGLEVEL:severity}* %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time} %{JAVACLASS:class}\.%{JAVAFILE:file}" }
}

}

and verified at grok patter varifier and it does not work.Any kind of help would be appreciated.

sysadmin1138 sysadmin1138 · Accepted Answer · 2017-02-06T14:59:44

You may find something like this works better:

^%{LOGLEVEL:security}%{SPACE}\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}\[%{DATA:appReqId}\]%{SPACE}\[%{DATA:AppUserId}\]%{SPACE}%{HOSTNAME:logger}:%{DATA:app_message}$

The insights here are:

Use %{SPACE} to handle one-or-more space instances, which can happen in some log formats. The * in your syntax can do that too, but this puts it more explicitly in the grok expression.
Use a dedicated timestamp format, %{TIMESTAMP_ISO8601} rather than attempt to break it apart and assemble later. This allows use of a date { match => [ "timestamp", ISO8601 ] } filter-block later to turn it into a real timestamp that will be useful in Kibana.
Capture the bracketed attributes directly in the grok expression.
Anchor the grok expression (the ^ and $ characters) to provide hints to the regex engine to make the expression less expensive to process.

Grok filter for application logs

1 Answers