Logstash grok multiple matches failure

Question

I have these logs where I'm trying to extract multiple information form each line with grok. But it seems some matches are not taken into account, although they seem fine to me.

Here's a sample line:

"#wtb# GetSP ok. Referer:http://test.redacted.com/path?query=string. Wtb:535e2554bdfdf33a22f564d0. Name:Client. Eans:3017565410073."

And heres' the related part of the conf file:

grok{
  break_on_match => false
  match => [
    "msg", "Referer:%{URI:referer}\.",
    "msg", "Wtb:%{WORD:wtb}",
    "msg", "Name:(?<name>[^\.]+)",
    "msg", "Eans:(?<eans>[\d,]+)",
    "referer", "https?://%{HOSTNAME:host}"
  ]
  tag_on_failure => []
}

I'm using the multiple matches because each line can be any combination of the various parameters given in sample.
In Kibana, the events have the referer and host fields added, but all others are missing (wtb, name, eans). I have no idea why. It does not stop after a successful match since the last pattern is added. Can anyone spot what I'm missing?

The syntax on the grok refence page worked for me with multiple matches. — Matthias

Alcanzar Alcanzar · Accepted Answer · 2015-04-15T17:58:45

The simple answer is that grok doesn't work like that. The way it's implemented, it only matches the first pattern in your grok list for a given key.

If you look at the source code of jls-grok-0.10.12/lib/grok/pure/pile.rb, you'll see this:

def match(string)
  @groks.each do |grok|
    match = grok.match(string)
    if match
      return [grok, match]
    end
  end
  return false
end # def match

which basically causes it to stop the grok pile for a given key after the first match.

So to do what you want, you are going to need to break your grok so that there's only one msg pattern per grok.

Logstash grok multiple matches failure

1 Answers