custom pattern to filter strings when using telegraf inputs.logparser.grok

Question

I'm trying filter for particular words in a log file using regex, the goal is that any log line that matches the regex in custom_pattern will go into influxdb, log lines that do not match willbe ignored. When I tested the regex it works, even in golang playground (https://play.golang.org/p/_apzOVwwgl2). But when I use it in the telegraf conf file as it is below, it doesn't work, there's no input into influxdb. Is there something I'm missing that should added to the configuration?

I've tested the regex on http://grokdebug.herokuapp.com/ and https://play.golang.org/p/_apzOVwwgl2 it works but not in the custom_patterns under [inputs.logparser.grok].

Here is my grok config

[[inputs.logparser]]
   files = ["/var/log/test1"]
   from_beginning = true

   [inputs.logparser.grok]
      patterns = ["%{FAIL_LOG}"]
      custom_patterns = '''FAIL_LOG ^.*?\b(multipathd?)\b.*?\b(failed|failing|(remaining active paths))\b.*?$'''

The pattern is supposed to match first 2 log lines like below and ignore the third line.

Oct 29 03:29:03 dc-as-5p multipath: checker failed interface 8:0 in map 150gb
Oct 29 03:29:03 dc-as-5p multipathd: checker failing interface 8:0 in map 150gb
Oct 29 03:26:03 dc-as-5p link: checker down remaining active paths interface 8:0 in map 150gb

What am I doing wrong?

When I changed the pattern to %{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME} (?<prog>multipathd?): (?<message>(.*(failed|failing|(remaining active paths)).*)) I was still able to match the first 2 log lines as expected, without matching the third log line. However, when I use this pattern inside inputs.logparser.grok, it still did not work, no input written to influxdb. Help PLEASE! — tfalade

NikNye NikNye · Accepted Answer · 2019-11-17T05:31:01

If you run telegraf with the --debug flag, you will see that it is having an issue parsing the logs.

$ telegraf --debug --config ./telegraf.conf
...
2019-11-17T05:01:07Z D! Grok no match found for: "Oct 29 03:29:03 dc-as-5p multipath: checker failed interface 8:0 in map 150gb"
2019-11-17T05:01:07Z D! Grok no match found for: "Oct 29 03:29:03 dc-as-5p multipathd: checker failing interface 8:0 in map 150gb value=3"
2019-11-17T05:01:07Z D! Grok no match found for: "Oct 29 03:26:03 dc-as-5p link: checker down remaining active paths interface 8:0 in map 150gb"

This error message is misleading because, as your testing has shown, your regex pattern is correct. The real issue is that you have not included a value to be logged in your regex.

A version of your regex to store the error message and timestamp might be:

    custom_patterns = '''FAIL_LOG %{SYSLOGTIMESTAMP:timestamp}.*(multipath).?: %{GREEDYDATA:message:string}'''

The value pattern can be found between ${}. Additional premade patterns can be found here. This will eliminate the first two errors above. The results of these can be seen using the --test flag.

$telegraf --test --config ./telegraf.conf
...
> logparser,host=pop-os,path=./test1 message="checker failed interface 8:0 in map 150gb",timestamp="Oct 29 03:29:03 " 1573968174161853621

For some reason the --test flag did not always output the results. I would have to run the command multiple times before getting the above output.

custom pattern to filter strings when using telegraf inputs.logparser.grok

2 Answers