Grok filter for log4js

Question

Iam trying to create a grok logstash filter for my log4js log.

The code in my nodejs app is as follows:

var httpLogFormat = ':remote-addr - - [:date] ":method :url ' + 'HTTP/:http-version" :status :res[content-length] ' + '":referrer" ":user-agent" :response-time';
log4js.loadAppender('file');
log4js.addAppender(log4js.appenders.file('logs/access.log'), 'access');
var logger = log4js.getLogger('access');
app.use(log4js.connectLogger(logger, { level: 'auto', format: httpLogFormat }));

This results in the following log message:

 [2017-01-31 08:54:32.491] [WARN] access - 192.1.1.10 - - [Tue, 31 Jan 2017 07:54:32 GMT] "GET /api/test HTTP/1.0" 304 undefined "https://localhost.com/test" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36" 111

My current grok filter looks like this (UPDATED):

grok {
     match => { "message" => "\[%{HTTPDATE:timestamp}\] \[%{WORD:loglevel}\] %{WORD:logtype} - %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \"%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})\" %{NUMBER:response} - \"%{DATA:rawrequest}\" \"%{QS:agent}\""}
   }

There is some parsing errors, and i suspect it is due to the [] but i'am unsure.

http://grokconstructor.appspot.com/ fails with:

NOT MATCHED. The longest regex prefix matching the beginning of this line is as follows: prefix " before match: [2017-01-31 08:54:32.491] [WARN] access - 192.1.1.10 - - [Tue, 31 Jan 2017 07:54:32 GMT] after match: GET /api/test HTTP/1.0" 304 undefined "https://test.localhost.com/test" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36" 111

Rumbles Rumbles · Accepted Answer · 2017-01-31T21:46:33

I've updated the grok to work for your example. I think you were misusing a few of the types (QS for example you don't need to have the "'s around it):

\[%{GREEDYDATA:timestamp}\]\ \[%{WORD:loglevel}\]\ %{WORD:logtype}\ -\ %{IPORHOST:clientip}\ %{USER:ident}\ %{USER:auth}\ \[%{GREEDYDATA}\]\ \"%{WORD:verb}\ %{NOTSPACE:request}(?: HTTP\/%{NUMBER:httpversion}|)\"\ %{NUMBER:response}\ %{WORD}\ \"%{DATA:rawrequest}\"\ %{QS:agent}\ %{INT:time_taken}

Check the docs for other words you can use.

Your parsing issues are probably down to literal use of the [ and ] characters as they are used in regex's, they need to be escaped as in my example.

Grok filter for log4js

1 Answers