Splunk regex line breaker, combining multiple ERROR [stderr] lines into one event

1
votes

I am working on displaying logs from docker json-file in splunk. For the most part I have it working, except for when a stack trace is printed. I know e.printStackTrace() isn't best practice, but our services have a few here and there that I want to support in Splunk.

I am configuring the props.conf file, and I have the following LINE_BREAKER regex in props.conf. LINE_BREAKER=([\n\r]+)\s*{"log":"[0-9]+.*[0-9]\s+

This will match up to the end of a timestamp of this form: {"log":"2019-04-18 15:18:18,796 ERROR [stderr] ...

The problem is that the printed stack trace is coming in multiple lines like this:

{"log":"2019-04-18 15:18:18,796 ERROR [stderr] (default-threads - 3) java.util.NoSuchElementException: No value present\n","stream":"stdout","time":"2019-04-18T19:18:18.800485539Z"}
{"log":"2019-04-18 15:18:18,804 ERROR [stderr] (default-threads - 3) at java.util.Optional.get(Optional.java:135)\n","stream":"stdout","time":"2019-04-18T19:18:18.806510971Z"}
{"log":"2019-04-18 15:18:19,259 ERROR [stderr] (default-threads - 3) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n","stream":"stdout","time":"2019-04-18T19:18:19.264689098Z"}
{"log":"2019-04-18 15:18:19,259 ERROR [stderr] (default-threads - 3) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n","stream":"stdout","time":"2019-04-18T19:18:19.281810119Z"}
{"log":"2019-04-18 15:18:19,274 ERROR [stderr] (default-threads - 3) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n","stream":"stdout","time":"2019-04-18T19:18:19.28185714Z"}
{"log":"2019-04-18 15:18:19,275 ERROR [stderr] (default-threads - 3) at java.lang.reflect.Method.invoke(Method.java:498)\n","stream":"stdout","time":"2019-04-18T19:18:19.281867696Z"}
{"log":"2019-04-18 15:18:19,275 ERROR [stderr] (default-threads - 3) at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49)\n","stream":"stdout","time":"2019-04-18T19:18:19.281875844Z"}
{"log":"2019-04-18 15:18:19,498 ERROR [org.apache.activemq.ActiveMQSession] (default-threads - 3) error dispatching message: : javax.ejb.EJBTransactionRolledbackException\n","stream":"stdout","time":"2019-04-18T19:18:19.523359025Z"}

... and so on. until the end of the stack trace. Each line ends up being an event based on my regex for LINE_BREAKER, which is fine for INFO messages and single line ERROR messages from our logger, but not for the stack trace as above.

I want to combine this stack trace of the ERROR [stderr] together into one event. So my line break should match until the next timestamp message that is not [stderr].

from my example, it should match from the first line with ERROR [stderr] until: {"log":"2019-04-18 15:18:19,498 (last log line above).

I have tried adding negative lookahead but now it will only match 1 time. ([\n\r]+)\s*{"log":"[0-9]+.*[0-9]\s+[\s\S]+\[(?!stderr]) matches until the last line, but if there are lines under that, none of them match anymore.

1
javaregexdockerloggingsplunk
You will have better luck using actual program logic to do this instead of regex. Anyway, I wouldn't take any time to post an answer for someone who's asked 23 questions and not accepted any answers. – miken32
This is to integrate with Splunk the output of our logs. Of course, it's much simpler with program logic. – user1738539

1 Answers

1
votes

In this part of the pattern {"log":"[0-9]+.*[0-9]\s+ you are matching the timestamp part using [0-9]+.*[0-9]\s+ can cause unnecessary bracktracking due to the .* which will first match until the end of the string.

You could make the timestamp like pattern more specific instead like [0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+

To get the matches, one option could be to match the line containing stderr. Then repeat matching the following lines with that pattern 0+ times starting with a newline to combine them.

To keep matching the other lines you could make use of a negative lookahead after ERROR and an alternation.

^(?:{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR \[stderr\].*(?:\n{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR \[stderr\].*)*|{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR (?!\[stderr\]).*)
  • ^ Start of the string
  • (?: Non capturing group
    • {"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR \[stderr\].*
    • (?: Non capturing group
      • \n{"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR \[stderr\].*
    • )* Close non capturing group and repeat 0+ times
    • | Or
    • {"log":"[0-9]{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+ ERROR (?!\[stderr\]).*
  • ) Close non capturing group

Regex demo