What is the logstash grok pattern for apache error..?

Question

I'm trying to configure a logstash filter for an apache log errors filter definition. It is basically the 'combined' LogFormat with some additional field, here is the apache log format definition:

[11446] [Thu Jan 30 07:50:49 2020] [debug]: RT::Date used Time::ParseDate to make '2020-01-31T07:20:46Z' 1580446800 (/app/rt4/sbin/../lib/RT/Date.pm:274)

or it could be of this format:

[Wed Jun 26 22:13:22 2013] [error] [client 10.10.10.100] PHP Fatal error: Uncaught exception '\Foo\Bar'

I tried the below solution:

APACHE_ERROR_TIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} APACHE_ERROR_LOG [%{APACHE_ERROR_TIME:timestamp}] [%{LOGLEVEL:loglevel}] (?:[client %{IPORHOST:clientip}] ){0,1}%{GREEDYDATA:errormsg}

but it gives a grok_match_failure.

Please help me with a generic working pattern for the above string (meaning it should not matter even if one of the part of string is not there)

Updated Question:

So, the above worked with https://stackoverflow.com/a/59986583/4127230 and I have again got stuck with another system log string.

Can you also suggest a generic grok pattern same for the below strings:

Feb 2 18:21:14 localhost prometheus_postgres_exporter: time=\"2020-02-02T18:21:14+04:00\" level=info msg=\"Error while closing non-pinging DB connection: \" source=\"postgres_exporter.go:1001\"

and

Feb 2 05:56:10 localhost logstash: [2020-02-02T05:56:10,934][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}

Why not use 2 different match cases for the same grok filter? — Sebastián Greco

baudsp baudsp · Accepted Answer · 2020-01-30T12:57:27

Using the provided APACHE_ERROR_TIME, I got a match with this pattern:

\[%{APACHE_ERROR_TIME:timestamp}\] \[%{LOGLEVEL:loglevel}\](?: \[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}

In addition to escaping the square brackets, there was some misplaced spaces that prevented the match.

For:

[11446] [Thu Jan 30 07:50:49 2020] [debug]: RT::Date used Time::ParseDate to make '2020-01-31T07:20:46Z' 1580446800 (/app/rt4/sbin/../lib/RT/Date.pm:274)

Result:

timestamp   Thu Jan 30 07:50:49 2020
loglevel    debug
clientip    
errormsg    : RT::Date used Time::ParseDate to make '2020-01-31T07:20:46Z' 1580446800 (/app/rt4/sbin/../lib/RT/Date.pm:274)
before match:   [11446]

For:

[Wed Jun 26 22:13:22 2013] [error] [client 10.10.10.100] PHP Fatal error: Uncaught exception '\Foo\Bar'

Result:

timestamp   Wed Jun 26 22:13:22 2013
loglevel    error
clientip    10.10.10.100
errormsg    PHP Fatal error: Uncaught exception '\Foo\Bar'

What is the logstash grok pattern for apache error..?

1 Answers