logstash for custom logs

0
votes

Im trying write a grok filter for below logs but getting grokfailure. I'm new to logstash, please help me.

Logs: msg.repository.routed.ABC_MAS:101::20170526-05:03:08: got from remote host at t-rate <0.068> and f-size <68> into tmp dir msg.repository.routed.ABC_MAS:101::20170526-05:03:07: got from remote host at t-rate <0.068> and f-size <68> into tmp dir msg.repository.routed.BCD_MAS:101::20170526-00:04:34: sftp connected to 1.2.2.1(msg), timeOut:1000 msg.repository.routed.ABC_MAS:101::20170526-00:04:37: sftp connected to 1.2.2.1(msg), timeOut:1000

Grok pattern which I used: filter { grok { match => { "message" => '(?: %{GREEDYDATA:pullpathinfo}):%{NUMBER:thread}::%{NUMBER:date}-%{NUMBER:HOUR}:%{NUMBER:MINUTE}:%{NUMBER:SECOND}: (?: sftp connected to %{IPORHOST:remoteip} %{GREEDYDATA:msg})' } match => { "message" => '(?: %{GREEDYDATA:pullpathinfo}):%{NUMBER:thread}::%{NUMBER:date}-%{NUMBER:HOUR}:%{NUMBER:MINUTE}:%{NUMBER:SECOND}: (?: got \<%{GREEDYDATA:filename}> %{GREEDYDATA:rate_size})' } } }

1
logstashlogstash-groklogstash-configuration

1 Answers

0
votes

To develop grok patterns I suggest you to use Grok Debugger. It allows you to build up the grok patterns incrementally.

For the follow log (on of the log lines in your question):

msg.repository.routed.ABC_MAS:101::20170526-00:04:37: sftp connected to 1.2.2.1(msg), timeOut:1000

the following grok pattern will work:

%{USERNAME:pullpathinfo}:%{NUMBER:thread}::%{NUMBER:date}-%{TIME:time}: sftp connected to %{IPORHOST:remoteip}%{GREEDYDATA:msg}

The follow changes are relevant:

  • The grok pattern has to be exact about every character in the pattern. This is also true for every space between the grok patterns (%{WORD} %{WORD} is not the same as %{WORD}%{WORD}. In your pattern there was a space too much between %{IPORHOST:remoteip} and %{GREEDYDATA:msg}.
  • %{USERNAME} instead of %{GREEDYDATA} (GREEDYDATA should only be used for remaining parts in a log line. Even if the pattern USERNAME has a name, which does not fit, the pattern behind it looks like a good fit, because it includes [a-zA-Z0-9._-]+ (but not the colon :)
  • %{TIME} instead of `%{NUMBER:HOUR}:%{NUMBER:MINUTE}:%{NUMBER:SECOND}

I hope this helps.